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Preface 


This  project  was  conceived  as  a  result  of  the  growing  realization  that 
the  Air  Force  project  manager  has  very  few  computerized  tools  with  which 
to  help  him  or  her  determine  which  alternative  projects  under  consideration 
for  further  development  should  be  pursued;  it  is  difficult  at  best  to  explain 
why  you  choose  a  certain  route,  or  alternative,  when  your  knowledge  and 
experience  tel)  you  it  ‘feels"  best.  A  computerized  decision  aid  that  helps 
the  project  manager  identify,  and  justify,  the  "best"  alternative  could  save 
many  of  the  dollars  and  careers  lost  in  the  past  due  to  incorrect  decisions. 

In  addition,  my  personal  experience  has  been  that  all  too  often  we  in  the 
Air  Force  speed  millions  of  dollars  on  software  development  when  a 
commercial  product  could  solve  the  problem  just  as  effectively. 

Thus,  with  an  distinct  application  in  mind,  and  with  the  objective  to 
locate  commercial  software  that  supports  that  application,  this  project 
took  shape:  To  locate,  acquire  and  test  a  commercial  microcomputer 
decision  aid  for  the  project  manager.  The  real  test  of  any  system  is  how 
well  it  performs  the  intended  job  in  actual  operation,  not  test  or  simulated 
operation.  The  test  data  is  a  integral,  and  invaluable,  part  of  this  project. 

This  research  could  not  have  been  performed  without  the  work  of  Captain 
Thomas  Triscari  and  Dr.  William  Henghold,  both  in  collecting  the  data  and 
guiding  me  through  the  pitfalls  of  academic  research;  my  thanks  to  both. 

But  the  most  appreciation  goes  to  my  loving  wife  Margaret  for  cheerfully 
enduring  and  supporting  this  task,  and  my  children  Phillip  II  and  Dawn;  they 
must  now  stop  calling  me  "Uncle  Daddy".  I  owe  my  family  a  debt  of  time  and 
attention  that  I  am  looking  forward  to  paying. 
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Abstract 


The  purpose  of  this  investigation  was  to  locate,  acquire,  and  test 
realistically  a  commercial  microcomputer  decision  support  system  (DSS) 
for  the  Air  Force  project  manager.  Only  commercial  microcomputer  DSSs 
were  considered  in  a  effort  to  demonstrate  the  cost  savings  that  could  be 
possible  by  using  off-the-shelf  software  for  a  dedicated  application  instead 
of  engaging  in  on  expensive  software  development  project. 

A  realistic  test  was  achieved  with  actual  project  data  collected 
specifically  for  a  detailed  project  selection  decision,  which  provided  a 
specific  utility  assessment,  and  by  managerial  use  of  the  DSS  which 
provided  a  general  utility  assessment. 

The  specific  utility  assessment  identified  some  limitations  with  this 
system.  The  general  utility  assessment  suggests  this  system  provides 
useful  managerial  decision  aid  in  many  decision  situations  involving 
predetermined  alternatives. 


A  COMMERCIAL  MICROCOMPUTER  DECISION  SUPPORT  SYSTEM  FOR  THE 
AIR  FORCE  RESEARCH  AND  DEVELOPMENT  PROJECT  MANAGER 


I.  Introduction 
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Background  and  General  Issue 

The  microcomputer  has  had  an  Impact  on  the  business  world  that  could 
not  have  been  predicted  ten  years  ago.  The  microcomputer  allows 
accountants  to  keep  better  records,  managers  to  more  effectively  manage 
budgets  and  schedules,  and  company  executives  to  better  gauge  successes 
and  failures.  Secretaries,  as  well  as  managers,  use  It  as  a  word  processor 
and  designers  use  It  as  a  drafting  tool.  Product  advertisements  and  reviews 
In  the  popular  computing  literature  Indicate  that  businessmen  are  using  the 
microcomputer  along  with  decision-assist  software  as  a  decision  aid,  but 
military,  and  specifically  Air  Force  research  and  development  (R&D) 
managers  are  not  mentioned  In  these  Journals.  However,  some  of  the 
commercial  software  may  be  as  applicable  to  military  applications  as  It  Is 
to  civilian  applications. 

Commercial  Software.  In  addition  to  the  billions  of  dollars  spent  on 
various  projects  and  project  management,  millions  of  dollars  are  spent  each 
year  by  the  Air  Force  on  software  development.  This  often  involves  the 
contracting  of  a  civilian  company  to  write  a  specialized  program  for  a 
single  Air  Force  application,  which  is  a  very  expensive  process.  Some  of 
this  development  money  could  be  saved  if  off-the-shelf  commercially 
available  programs  were  used  for  the  Air  Force  application  rather  than 
engaging  In  expensive  software  development  contracts.  Commercial 
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software  that  aids  the  program  manager’s  project  selection  decision 
process  could  provide  a  savings  in  both  the  development  of  the  software  and 
in  an  enhanced  selection  process  for  future  projects. 

Project  Selection  and  Decision  Support  Systems  (DSS).  The  Air  Force 


directs  and  manages  R&D  projects  that  involve  many  new  and  different 
technologies.  Each  R&D  project  manager,  working  with  limited  resources, 
must  use  his  or  her  own  methods  to  choose  the  most  promising  prospects 
among  the  relevant  technologies.  Some  projects  are  significantly  promising 
and  lead  directly  to  a  clear  choice  among  many  possibilities.  Other 
projects,  however,  require  an  additional  and  more  In-depth  evaluation,  or 
trade-off,  based  on  some  criteria  for  developing  a  most  appropriate 
Investment  strategy.  This  decision  process  can  be  described  as 
"semi-structured"  since  it  often  requires  the  manager  to  make  an  intuitive 
decision  about  what  technology  or  research  effort  to  pursue.  Donnelly  et  al 
point  out  that  "resources  (financial  and  nonfinancial)  are  becoming  more  and 
more  scarce,  in  such  a  situation,  choices  must  be  made,  and  some  method  is 
necessary  to  help  management  make  the  choices"  (7: 140).  The  realities  of 
normal  budget  constraints  faced  by  every  manager  plus  the  additional 
constraints  forced  on  the  military  manager  due  to  such  things  as 
Gramm -Rudman  impacts,  make  each  decision,  or  lack  thereof,  even  more 
critical.  A  decision  aid  for  the  military  project  manager  could  help  to  avoid 
mistakes  and  add  structure  to  the  semi-structured  decision  process.  A 
computer-based  decision  aid,  or  DSS,  could  further  assist  the  project 
manager  by  providing  a  structured  decision  format  which  aids  the  transition 
from  a  semi -structured  to  a  structured  decision  process.  In  addition,  a 
computer-based  DSS  could  provide  results,  such  as  graphical  outputs,  not 


readily  available  with  pencil  and  paper  analysis. 

Microcomputer  DSSs.  While  DSSs  can  be  quite  complex  in  terms  of 
hardware  and  software,  there  are  three  primary  reasons  that  only 
microcomputer  DSSs  are  considered  for  this  project.  The  first  reason  for 
this  is  that  time,  money,  equipment,  and  manpower  are  limited;  the 
resources  are  not  available  for  a  long-term,  expensive 
search-purchase-evaluation  project.  Second,  a  microcomputer  DSS  can 
provide  a  personal,  dedicated,  and  often  private  environment  for  managerial 
decision  making.  This  includes  the  fact  that  microcomputer  programs  are 
transportable  and  therefore  provide  the  greatest  potential  for  use  at  any 
office  or  laboratory  in  the  Air  Force.  A  terminal-modem-mainframe 
computer  link  could  provide  personal  DSS  service  but  this  would  rarely  be 
dedicated  (the  manager  could  wait  for  computer  time  or  be  "bumped  off" 
(Kring  use)  and,  despite  security  precautions,  would  never  be  completely 
private  from  other  computer  users.  Therefore,  the  same  microcomputer  DSS 
could  provide  a  familiar  and  dedicated  decision  aid  at  almost  any  permanent, 
or  temporary,  duty  location.  And  third,  the  specific  microcomputer  used  for 
this  research,  an  Apple  Macintosh*,  best  satisfies  one  of  the  primary 
system  requirements  (discussed  In  Chapter  III):  A  managerial  DSS  must 
provide  an  exceptional  ease  of  use  for  Infrequent,  non-technical,  users  of 
the  system.  This  Is  accomplished  with  a  graphic  display/mouse  input  that 
provides  a  less  intimidating  user  Interface  which  may  encourage  more 
frequent  managerial  use. 

As  noted  above,  DSSs  can  be  quite  complex.  However,  simple  rank 
ordering  of  numeric  project  data  can  be  accomplished  with  a  spreadsheet 

*  Apple  is  a  licensed  trademark  of  Apple  Computer,  Inc. 

Macintosh  Is  a  licensed  trademark  of  Apple  Computer,  Inc. 


type  of  program.  A  DSS  can  take  this  process  a  step  closer  to  facilitating 
semi -structured  decision  making  by  supporting  visual  decision  model 
building  (graphically  depicting  the  decision  model),  allowing  normally 
non-quantifiable  inputs  in  the  decision  model  (inputs  within  a 
semi -structured  range  such  as  good  to  bad  or  like  to  dislike)  and  by 
providing  a  graphic  output.  DSS  capabilities  are  discussed  in  Chapter  III. 

One  final,  noteworthy,  benefit  of  using  a  DSS  Is  that  it  can  provide  a 
computer  printout  which  is  a  permanent  record  of  the  decision  process.  The 
printout  can  be  included  in  reports  as  a  justification  and  explanation  and  as 
an  after  the  fact  audit  trail  if  a  question  is  later  raised  about  the  decision. 


Specific  Problem 

The  general  Issue  discussed  above  leads  directly  to  the  following 
research  question:  What  is  the  feasibility  of  structuring  and  aiding  the 
semi-structured,  or  intuitive,  R&D  project  selection  decision  process 
using  a  commercial  microcomputer  decision  support  system?  The  process 
of  structuring  and  aiding  a  semi-structured  R&D  decision  using  a 
commercial  microcomputer  DSS  involves  two  specific  problems: 

1.  How  are  semi-structured  R&D  project  decisions  made  now?  What 
Intuitive  decision  processes  do  project  managers  use  in  conjunction  with 
structured  decision  processes  to  assign  program  priorities  and  identify  the 
most  promising  technologies  ?  This  can  be  answered  from  past  research 
and  is  discussed  In  Chapter  III. 

2.  How  can  a  commercial  microcomputer  DSS  add  structure  to  and  aid 
the  semi -structured  decision  process  ? 


Investigative  Questions 

The  discussion  of  the  general  issue  resulted  In  two  specific  problems; 
two  investigative  questions  must  be  answered  to  resolve  the  two  specific 
problems: 

1.  What  are  the  semi-structured  decision  processes  that  Air  Force  R&D 
project  managers  use  now  when  assigning  program  priorities? 

2.  What  commercial  microcomputer  decision  aid  best  fits  the  specific 
OSS  criteria  and  will  aid  this  R&D  semi -structured  project  selection 
decision  process? 


Research  Objective 

This  research  Is  focused  on  a  distinct  objective:  To  locate,  acquire, 
demonstrate  realistically,  and  assess  a  commercial  microcomputer  DSS  that 
supports  and  aids  the  R&D  project  manager's  semi-structured  project 
selection  decision  process.  The  DSS  location  and  assessment  criteria  and 
the  DSS  selection  process  are  In  Chapter  III. 

Project  Data.  The  best  method  to  realistically  demonstrate  the 
operation  of  a  DSS  Intended  to  support  the  project  selection  decision 
process  Is  to  perform  project  selections  with  data  derived  from  actual 
projects  that  will  proceeding  in  the  near  future.  Therefore,  the  data  used 
for  this  test  are  from  the  Research  Planning  Workshop  for  Artificial 
Intelligence  In  Manufacturing  conducted  31  July  1985  to  1  August  1985  (12). 
The  workshop  data  are  described  In  Chapter  III  and  the  utility  analysis  of 
the  DSS/data  combination  is  in  Chapter  IV. 

Managerial  Feedback.  The  primary  assessment  of  this  DSS  Is  a  result  of 
exercising  the  system  with  the  data  and  is  in  Chapter  IV,  Utility  Analysis. 


However,  the  Intended  users  are  managers  and  their  perceptions  of  this 
system  are  paramount.  Therefore,  managerial  assessment  of  this  D55  is 
provided  through  feedback,  written  and  verba),  from  managers  that 
participated  in  a  “hands-on"  demonstration  of  the  DSS.  This  consisted  of  a 
brief  demonstration  of  the  DSS  followed  by  the  opportunity  for  each 
manager  to  manipulate  the  DSS  as  If  he  or  she  were  performing  the  project 
selection.  Each  manager-participant  filled  out  a  questionnaire  at  the  end  of 
the  session  indicating  their  perceptions  of  the  DSS.  The  questionnaire  is 
developed  in  Chapter  Hi  and  the  results  are  discussed  in  Chapter  V. 


II.  Methodology 

Introduction 

This  chapter  discusses  the  research  methodology  necessary  to  locate, 
acquire,  realistically  demonstrate,  and  assess  this  microcomputer  DSS. 
First,  the  scope  and  limitations  of  the  research  are  discussed  to  set  the 
stage  for  the  steps  that  follow.  Then  the  specific  steps  required  for  this 
research  are  discussed  in  the  section  titled  Particular  Method.  The  research 
flow  Is  Illustrated  In  Figure  1. 


Managerial 
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Commercially 
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Available  OSS 
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Flgure  1.  DSS  Research  Methodology/DSS  Choice  Flow  Chart 

The  steps  (Indicated  in  Figure  1 )  follow  a  sequential  flow,  the  type  of 
decision  to  t>e  supported  Is  Identified  first,  then  the  applicability  of  DSSs  to 
that  decision  type  On  general)  is  determined,  then  the 
avallablllty/appllcablllty  of  commercial  microcomputer  DSSs,  which  are 
further  limited  by  additional  constraints,  Is  determined,  and  finally  the 
actual  utility  analysis  and  managerial  assessment  of  the  DSS  Is 
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accomplished. 

Scope 

The  research  objective  is  to  locate,  acquire,  realistically  demonstrate, 
and  assess  a  commercial  microcomputer  DSS  that  supports  and  aids  the  Air 
Force  R&D  project  manager's  semi -structured  decision  process.  Thus,  the 
scope  of  this  research  is  limited  to  0 )  Identifying  a  semi-structured 
decision  process  used  by  Air  Force  R&D  laboratory  managers  in  R&D  project 
evaluation,  termed  the  decision  environment,  through  a  literature  review, 

(2)  Identifying  commercial  microcomputer  DS5s  through  computer  literature 


and  product  advertisements  that  support  this  R&D  decision  environment, 

(3)  Acquiring  the  one  DSS  that  best  fits  developed  criteria,  (4)  Building 
decision  models  with  the  DSS  and  performing  a  decision  analysis  with 
actual  data,  (5)  Presenting  the  DSS  to  a  sample  of  project  managers  for 
their  assessment,  and  (6)  Analysis  and  discussion  of  the  utility  analysis, 
the  results,  and  the  impressions  that  the  DSS  made  on  the  managers. 

Limitations.  As  discussed  later  in  the  section  called  Additional 
Constraints,  the  constraints  of  limited  time,  limited  money,  and  of  using  a 
Macintosh  microcomputer  for  full-time  DSS  research  limit  the  scope  of  this 
research  project.  These  constraints  limit  the  number  of  DSSs  available  for 
this  research  and  possibly  decrease  the  utility  of  the  DSS  as  a  decision  aid.  ^ 

The  utility  of  the  DSS  Is  discussed  in  Chapter  V,  after  the  utility  analysis  Is 
performed  In  Chapter  IV. 

This  research  is  not  directed  toward  development  of  a  DSS  of  such  broad 
utility  that  It  supports  every  'what  if"  situation  or  every  conceivable  report 
and  data  format.  For  example,  a  decision  aid  that  handles  a  "what  if" 
analysis  involving  10  or  20  alternatives  with  40  or  50  factors  each,  may  not 
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be  affordable  and/or  may  not  provide  an  acceptable  graphics  capability  or  a 
user  friendly  Input/output  format;  a  DS5  that  performs  evaluations  of  only 
five  alternatives  with  six  to  ten  factors  each  may  be  more  affordable, 
provide  a  better  graphics  capability,  and  a  better  user  interface.  Similarly, 
all  microcomputer  DSSs  may  not  support  ad  hoc  report  requests  such  as  for 
’A  List  of  every  alternative  with  a  Criteria  3  rank  above  5“.  In  either  case, 
for  some  decisions,  it  may  be  necessary  to  invest  a  considerable  amount  of 
time  in  data  preparation  and/or  data  entry  into  the  DSS  which  would  result 
in  more  difficult  DSS  preparation  for  use.  Thus  the  data  interface  could 
require  the  DSS  to  be  prepared  for  use  by  someone  familiar  with  it  prior  to 
turning  it  over  to  the  manager  who  would  then  manipulate  it  for  the  decision 
results. 

Particular  Method 

The  Decision  Environment.  Past  research  on  decision  methods  and  R&D 
project  selection  methods  provides  the  starting  point  for  this  research.  The 
research  by  Lee  ( 1 1 )  and  Prince  05)  was  specifically  aimed  at 
identification  of  semi -structured  project  selection  decision  processes  and 
is  used  to  develop  the  framework  of  a  representative  semi-structured 
decision  process  for  the  Air  Force  R&D  laboratory  management  environment. 
This  framework  establishes  the  decision  environment  and  is  used  along  with 
the  DSS  criteria  to  judge  the  appropriateness  of  the  DSSs  that  are  located. 
The  decision  environment  is  discussed  in  Chapter  ill. 

DSS  Identification.  Next,  a  “system"  is  discussed  and  some  of  the  ideal 
DSS  criteria  are  identified  from  the  literature  reviewed.  The  ideal  DSS 
criteria  are  further  broken  down  Into  specific  criteria.  Then,  for  the  DSS 


selection  process,  some  microcomputer  DSSs  are  located  through  product 
reviews,  advertisements,  and  product  Information  from  the  manufacturers. 
The  number  of  applicable  microcomputer  DSSs  Is  first  reduced  by  limiting 
them  to  only  those  that  fit  the  project  selection  decision  environment  as 
opposed  to  those  DSSs  oriented  toward  alternative  generation,  personnel 
evaluation  or  product-sales  decisions.  The  refined  DSS  criteria  are  used  in 
conjunction  with  the  additional  constraints  discussed  below  for  the 
selection  of  one  of  the  remaining  commercial  microcomputer  DSSs.  The  DSS 
review  and  comparison  is  reported  in  Chapter  III. 

Additional  Constraints.  Not  only  must  the  DSS  chosen  for  this  project 
meet  the  requirements  discussed  above  and  detailed  in  Chapter  III,  it  must 
also  meet  some  strict  research  constraints.  These  constraints  and  their 
reasons  are: 

1 .  The  DSS  must  be  obtained  and  useable  before  3 1  April  1 986  so  that 
AFIT  deadlines  for  this  research  can  be  met 

2.  The  DSS  must  be  bought  for  under  $  100  since  there  is  no  research 
money  available  and  personal  money  must  be  used  if  purchase  becomes 
necessary. 

3.  The  DSS  will  operate  on  an  Apple  Macintosh  computer  since  it  is  the 
only  microcomputer  available  for  full-time  DSS  research,  and  it  fits  many 
of  the  system  requirements  discussed  in  Chapter  III. 

Realistic  DSS  Demonstration.  The  last  steps  in  this  project  are  a 
realistic  demonstration,  or  utility  analysis,  and  a  managerial  assessment  of 
the  DSS.  The  DSS  is  used  to  build  a  specific  decision  model  called  the 
decision  environment  model  (DEM).  The  utility  analysis  of  this  DSS  involves 
building  a  DEM  for  each  of  the  three  groups  of  projects  generated  by  the 


Research  Planning  Workshop  for  Artificial  Intelligence  (Al)  in 
Manufacturing.  The  Al  project  data  is  discussed  in  Chapter  III  and  the 
project  models  are  built  and  exercised  with  the  data  in  Chapter  IV.  After 
the  models  are  built,  the  scoring  data  entered  and  the  top  five  projects  in 
each  group  indicated  by  the  DSS,  the  weight  (or  priority)  of  a  criterion  is 
shifted  to  see  what  change  in  project  selection  occurs  due  to  the  shift,  or 
’what  if"  the  weight  is  different.  Therefore,  the  the  majority  of  the 
assessment  of  how  well  this  DSS  fits  the  specific  decision  situation  is  as  a 
result  of  exercising  the  system  (Including  initial  training,  model  building, 
data  preparation  and  entry,  and  sensitivity  analysis)  for  this  research 
project  instead  of  from  the  short  time  that  managers  use  the  system  and 
provide  subsequent  feedback.  But  after  the  utility  analysis  is  the 
managerial  assessment  of  this  DSS. 

Managerial  Assessment.  The  managerial  assessment  is  considered 
“hands-on"  because  first  the  DSS  is  demonstrated  and  then  each  manager  is 
allowed  to  operate  the  DSS  in  an  actual  decision  situation.  Since  managers 
participate  in  a  decision  (or  managerial  utility)  analysis  this  will  give  a 
test  of  ease  of  use  for  the  managers  since  they  are  not  involved  in  building 
the  DEMs  or  entering  the  project  data.  Also,  this  may  give  an  indication  of 
the  training  required  for  managers  that  may  use  the  DSS  infrequently.  The 
assessment  of  the  DSS  is  provided  by  managerial  feedback. 

The  managerial  feedback  is  achieved  through  a  short  questionnaire  that 
was  completed  by  each  manager  that  participated  in  the  DSS  hands-on 
demonstration.  In  addition,  some  of  the  verbal  comments  from  the 
demonstrations  are  included  in  the  results  and  discussed.  The  questionnaire 
is  developed  in  Chapter  III  and  the  results  are  discussed  in  Chapter  V. 


III.  Literature  Review  and  Preliminary  Research 


Introduction 

This  literature  review  begins  with  a  short  discussion  of  decision 
processes.  The  discussion  then  progresses  to  previous  research  performed 
about  the  R&D  project  selection  decision  process  in  general  and  specifically 
that  process  used  at  the  Air  Force  Wright  Aeronautical  Laboratories,  which 
establishes  the  decision  environment.  Then  the  composition  of  a  system  is 
discussed  and  the  broad  area  of  decision  support  systems  is  reviewed.  This 
includes  how  some  DSSs  work  and  the  continuing  controversy  over  the  user 
Interface.  Next,  a  brief  survey  of  some  operational  DSSs  is  conducted  and 
the  field  of  DSS  software  is  narrowed  down  to  that  now  commercially 
available  for  microcomputers.  The  DSS  criteria  for  this  research  are  then 
derived  from  the  literature  and  the  additional  constraints  discussed  In 
Chapter  II.  These  criteria/restraints  are  refined  into  a  software  selection 
matrix  which  is  used  for  selection  of  the  Macintosh  DSS  software  that  best 
supports  the  decision  environment  and  meets  the  additional  constraints. 
Next  is  a  section  discussing  the  project  data  with  which  this  DSS  Is  tested. 
Last  Is  a  section  on  the  development  of  the  Management  Questionnaire  which 
Is  the  instrument  used  for  the  managerial  assessment  of  this  DSS. 

Although  many  of  the  sections  introduced  above  are  not  part  of  the 
literature,  they  are  necessary  for  completion  of  this  research  and  they 
evolve  from  the  literature  in  a  natural  sequence.  Therefore,  those  sections 
that  comprise  part  of  this  research  process  are  Included  with  the  literature 
to  preserve  the  flow  of  thought  and  for  clarity. 
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The  Decision  Process 

Almost  my  textbook  on  monogement  describes  the  most  recognized 
decision  processes.  Keen  end  Morton  Identify  five  views  on  decisionmaking. 


1.  The  economic rational  concept :  This  is  the  clessfcel 
normative  theory  of  decisionmaking,  in  which 
decisionmakers  are  all-knowing  and  able  to  evatuote  all 
alternatives.  They  are  dissatisfied  with  any  solution  but 
the  best. 

2.  The  satisficing,  process-oriented  view,  this 
considers  the  decisionmakers  to  be  Intendedlu  rational 
although  cognitive  limits  lead  to  a  bounded  rationality; 
thus  the  goal  of  any  decisionmaker  is  to  get  a  good 
enough  answer,  not  the  best  possible  one.  This  point  of 
view  stresses  the  process  of  decisionmaking  and  not 
just  its  outputs;  it  emphoslzes  the  relatively  limited 
analysis  and  search  most  managers  will  make  and  their 
reliance  on  heuristics. 

3.  The  organizations} procedures  view  this  focuses  on 
the  interrelations  among  components  of  the  organization. 
It  highlights  organizational  structure,  mechanisms  for 
communication  and  coordination,  and  the  standard 
operating  procedures  by  which  decisionmaking  is 
systematized  and  often  simplified. 

4.  The  politico)  view,  this  regards  the  participants  in 
the  decision  process  os  actors  with  ports  to  ploy.  They 
have  strong  Individual  preferences  and  vested  Interests 
and  form  coalitions  of  organizational  subgroups. 
Decisions  are  frequently  dominated  by  bargaining  and 
conflict,  with  the  result  that  only  small  deviations  from 
the  status  quo  are  normally  possible.  Major  innovations 
ore  (quite  reasonably)  resisted  by  those  whose  position, 
interests,  or  simply  job  satisfaction  will  be  affected. 

5.  The  individual  differences  approach-,  this  view  argues 
that  on  Individual's  personality  and  style  strongly 
determine  his  or  her  choices  and  behavior.  Personal 
“rational ity"  Is  subjective  ond  behovior  is  very  much 
determined  by  the  manner  in  which  an  individual 
processes  information.  [9:80] 


These  decision  methods  or  schools  of  thought  were  developed  through 
research  on  civilian  management.  The  methods  used  by  Air  Force  R&D 
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managers  appear  to  be  different  but  in  fact  fit  into  one  or  more  of  the 
categories  listed  above. 

The  Project  Selection  Decision  Process 

Dr.  David  Lee  conducted  research  for  development  of  a  decision  aid  for 
the  MANTECH  project  selection  process.  He  found  that  "MANTECH  project 
selection  and  resource  allocation  decisions  are  complex  mental  processes 
involving  numerous  considerations.  Without  a  systematic  evaluation 
methodology,  some  considerations  can  be  overlooked,  while  others  may  be 
overemphasized"  01:13).  His  response  was  to  propose  project  score  and 
summary  sheets  to  add  consistency  to  the  project  selection  process  (11:13). 
These  score  and  summary  sheets  are,  in  fact,  one  form  of  a  DSS;  any  plan 
that  aids  the  decision  process  could  be  called  a  D55.  It  is  not  necessary  for 
the  plan  to  operate  on  a  computer  to  qualify  as  a  DSS,  but,  as  discussed 
previously,  a  computer-based  decision  aid  could  provide  many  benefits  not 
readily  available  through  other  analysis  techniques. 

Captain  Jeremy  Prince  performed  research  Into  the  methods  used  for 
R&D  project  selection  at  the  Wright  Aeronautical  Labs.  He  found  that  the 
three  main  factors  used  are  Air  Force  need,  technical  merit,  and  resource 
availability  (15:37).  in  addition,  Captain  Prince  found  that  no  formal 
decision  method  was  used;  decisions  were  made  "via  a  group  consensus  of 
researchers  or  by  a  management  committee",  but  that  the  most  used  formal 
decision  techniques  were  checklist  and  profile  charts,  scoring  models, 
decision  trees,  and  goal  programming  (15:38).  This  would  indicate  a 
satisficing  approach  to  Air  Force  R&D  project  decisionmaking,  and  it  is 
often  Influenced  by  political  and  organizational  pressures.  Prince 
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recommends  further  research  into  the  "impact  a  decision  support  system 
may  have  on  research  project  selection"  (15:42).  Therefore,  a  DSS  that  aids 
the  project  selection  decision  process  should  provide,  or  in  some  way 
support,  one  of  those  techniques  Prince  found  used  most  often:  Checklists 
and  profile  charts,  scoring  models,  decision  trees,  end/or  goal  programming. 
However,  "The  impact  of  the  DSS  is  often  qualitative;  it  does  not 
necessarily  reduce  costs  or  directly  increase  profits  but  "improves"  the 
decision  process"  (9:99).  Thus,  one  method  for  improving  the  Air  Force  R&D 
project  selection  decision  process  would  be  to  assist  the  project  manager 
with  one  of  the  techniques  identified  by  Prince. 

The  Decision  Environment.  The  decision  environment  includes  the 
decision  maker's  organization,  the  decision  maker,  and  the  type  of  decision 
as  illustrated  in  Figure  2  from  Adelman  (1:335).  The  interfaces  are  depicted 
by  the  sets  of  arrows  and  the  Decision  Making  Organization  interface  refers 
"to  what  extent  the  DSS  facilitates  the  decisionmaking  process  of  the 
organization"  (1:335).  The  interface  between  the  user  and  the  DSS  indicates 
not  only  that  the  DSS  should  be  userfriendly,  providing  information  in  and 
out  in  a  format  that  the  decision  maker  finds  useful,  but  that  the  DSS  should 
support  the  type  of  decision  that  the  decision  maker  faces.  In  addition,  the 


DSS  should  support  a  recognizable  (by  the  decision  maker)  decision  process. 
Therefore,  a  DSS  designed  to  aid  an  R&D  project  manager  should  work  within 
his  or  her  decision  environment.  In  other  words,  does  the  DSS  support  only 
business  related  topics  such  as  sales,  marketing  and  personnel  or  will  it 
support  the  Air  Force  R&D  project  manager's  decision  environment  which 
involves  semi -structured  decisions  concerning  new  technologies  and  project 
alternatives? 
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Figure  2.  Pictorial  framework  for  considering  Issues  relevant  to 
the  design  and  evaluation  of  Decision  Support  Systems  [1: 335] 


Decision  Support  Systems 

Systems.  Before  a  DSS  can  be  assembled  to  support  a  specific 
application,  the  general  concept  of  a  "system*  must  be  explored.  Donnelly  et 
al.  say  that  “A  system  Is  a  collection  of  objects  united  by  some  form  of 
regular  Interaction  and  interdependence"  (7:544).  "Each  system  is  composed 
of  subsystems  which  In  turn  are  made  up  of  other  subsystems*  (4:271). 
Subsystems,  then,  would  be  the  objects  united  but  some  subsystems  may 
only  be  comprised  of  individual  components.  In  either  case,  "The 
interconnections  and  Interactions  between  the  subsystems  are  termed 
Interfaces*  (4271). 

Subsystems.  The  components  of  the  system,  the  subsystems,  will 
determine  the  system.  For  a  DSS,  the  system  may  only  be  comprised  of  the 
hardware  subsystem  and  software  subsystem.  The  hardware  could  be  as 


large  as  a  mainframe  computer  or  as  small  as  a  microcomputer,  and  In  some 
cases  may  only  be  a  coin  to  flip.  However,  a  DS5  In  the  context  of  this 
research  effort  Involves  a  computer  system.  A  computer  system  requires 
both  an  operation  software  subsystem  and  an  application  software 
subsystem,  at  a  minimum,  to  perform  a  useful  function.  The  operation 
software  subsystem  runs  the  computer  and  provides  the  user  interface,  in 
conjunction  with  the  particular  type  of  computer,  while  the  application 
software  subsystem  is  what  performs  the  work,  in  this  case  the  “decision 
support’  in  DSS.  The  DSS  for  this  research  will  be  composed  of  a 
microcomputer  subsystem  and  a  software  subsystem,  but  it  must  Interface 
with  the  user  and  the  data.  The  user  interface  and  the  data  will  be 
discussed  more  later. 

Training,  implied  in  the  use  of  a  system  is  the  training  required  for  the 
user  to  effectively  use  it.  Since  this  DSS  project  is  focused  on  a 
userfrlendly  managerial  system,  training  should  be  kept  to  a  minimum.  The 
result  of  providing  minimum  training  to  the  manager/user  becomes  evident 
through  their  subsequent  use  and  assessment  of  the  system  and  is  discussed 
in  Chapter  V. 

DSS.  Thousands  of  computer  programs  have  been  written  to  ease  the 
burden  of  almost  every  repetitive  or  complex  task.  But  only  within  the  past 
two  or  three  years  have  microcomputers  become  powerful  enough  to  run 
programs  to  help  make  decisions.  “DSSs ...  are  designed  specifically  to 
support  particular  decision  processes  rather  than  to  expedite  and/or 
automate  transaction  processing,  record  keeping,  and  normal  business 
reporting’  (10:17).  Davis  identifies  three  characteristics  that  a  DSS  should 
have  to  be  effective.  These  are: 


1 .  The  computer  must  support  the  manager  but  not 
replace  his  or  her  judgment.  It  should  therefore  neither 
try  to  provide  the  answers'  nor  Impose  a  predefined 
sequence  of  analysis. 

2.  The  main  payoff  of  computer  support  is  for  semistructured 
problems,  where  parts  of  the  analysis  can  be  systematized 
for  the  computer,  but  where  the  decision  maker  s  insight 

and  judgment  are  needed  to  control  the  process. 

3.  Effective  problem  solving  Is  interactive  and  is  enhanced 
by  a  dialogue  between  the  user  and  the  system.  The  user 
explores  tne  problem  situation  using  the  analytic  and 
information-providing  capabilities  of  the  system  as  well 
as  human  experience  and  Insights.  [4:368-3691 


Davis’  three  characteristics  establish  a  broad  framework  within  which  a 
DSS  should  operate.  For  a  DSS  to  exhibit  the  characteristics  that  Davis 
Identifies,  It  should  perform  some  specific  functions  that  in  combination 
form  the  characteristics  of  the  DSS.  Kosy  and  Dahr  derive  some  suggested 
DSS  functions  in  the  following  way. 


Staff  activities  can  be  supported  in  many  ways  and 
the  types  of  functions  served  by  DSS  cover  a  wide  spectrum 
They  range  from  those  that  are  very  data-oriented  to  those 
that  are  very  model-oriented.  The  following  points  on  this 
spectrum  may  be  identified: 


-  Retrieve  data  items,  consolidate,  and 
prepare  prescribed  reports 

(eg.,  roll  up  a  resource  plan) 

-  Provide  tools  for  ad  hoc  data  analysis 
(e.g.,  time-series  extrapolation) 

-  Calculate  consequences  of  alternative 
decisions  (e.g.  simulation) 

-  Generate  and  suggest  good  or  optimal 
decisions  in  specialized  domains 
(e^.,  Inventory  optimization) 


However,  in  addition  to,  and  sometimes  instead  of,  the  functions  that  Kosy 
and  Dahr  recommend,  some  other  functions  for  a  D5S  to  be  adequate  could  be 
added.  "It  should  provide  a  framework  in  which  all  available  information  is 
used  to  deduce  which  of  the  decision  alternatives  is  “best"  according  to  the 
decision  maker's  preferences"  (13:200).  In  addition,  "features  may  include 
curve  fitting,  Monte  Carlo  (probabilistic)  solutions,  and  goal  seeking" 

(22:77)  which  in  simpler  terms  mean  fitting  a  curve  to  data  points  to 
facilitate  trend  analysis,  using  simulation  techniques  to  determine  the 
probability  of  an  event  occurring,  and  changing  the  output  to  a  desired  level 
to  evaluate  the  effect  on  the  input  data.  A  DSS  should  include  "speed  of 
response  to  maintain  the  manager’s  own  thought  processes;  communication 
with  the  manager  in  terms  familiar  with  him;  and  a  structure  which  is 
understood  by  the  manager'  (22:78).  Once  a  model  Is  built  or  a  scenario 
established,  what-lf  changes  to  the  model  can  be  tested  for  their  effect  on 
the  outcome  (22:77).  In  total,  the  DSS  is  a  hardware-software  combination 
that  will  aid  the  decision  maker  in  the  decision  process  by  quickly 
processing  and  presenting  data  using  DSS  techniques.  The  techniques  for 
processing  and  presenting  the  data  are  the  heart  of  DSSs. 

Some  DSS  Analysis  Techniques.  The  data  processing,  or  analysis, 
function  of  DSS  can  be  performed  either  by  assigning  preferences,  which  is 
called  utility  theory  (13:201),  or  by  probability  theory  (13:205).  In  either 
case  the  decision  model,  a  decision  tree,  is  weighted  by  the  preferences  or 
probabilities  assigned  to  each  decision  branch  on  the  tree.  Figure  3  is  a 
simple  decision  tree  which  illustrates  two  possible  choices  at  each  branch. 
A  DSS  would  calculate  the  probability  of  A(  1 )  occurring  as  0.5  multiplied 
times  0.7  or  0.35. 
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The  tree  is  much  more  complicated  in  practice  and  in  addition,  most  DSS 
users  have  no  knowledge  of  the  numbers,  where  they  come  from  or  how  the 
DSS  works  to  provide  a  result  ( 1 0:21 ).  Such  a  lack  of  knowledge  could 
result  in  improper  DSS  operation  and  therefore  erroneous  decision  results. 
Some  of  these  deficiencies  are  overcome  by  the  use  of  a  knowledge-based 
support  system,  a  concept  which  comes  from  research  in  the  field  of 
artificial  intelligence  and  allows  the  computer  "to  store,  apply,  and 
communicate  knowledge*  ( 10:21 ).  The  state-of-the-art  of  microcomputers 
and  their  programs  has  reached  the  level  within  the  past  year  to  use  Al 
techniques  but  they  are  not  yet  in  wide  use. 

The  User  Interface.  For  any  managerial  computer  system  to  be  used,  it 
should  be  easy  to  learn  and  to  use.  The  benefit  or  service  that  it  provides  is 
almost  secondary  when  weeks  or  months  are  spent  learning  the  system  and 
the  manager  must  relearn  it  each  time  the  system  is  changed  or  not  used  for 
a  few  days.  "This  is  the  province  of  the  user  interface.  To  eliminate  the 
need  for  a  computer  specialist,  the  interface  should  be  flexible,  forgiving, 
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fast,  informative,  and  easy  to  learn  without  training"  (10:24).  Reimann 
states  that  "ease  of  use  or  "user  friendliness"  is  perhaps  the  single  most 
vital  criterion"  (17:23).  Userfriendly  criteria  dictates  that  "the  design  of 
the  DSS  must  be  focused  on  potential  users  and  their  decisionmaking  needs" 
(3:457). 

An  important  part  of  userfriendliness  involves  the  method  of 
communication  with  the  computer.  "Explicit  communication  may  involve 
specific  displays  and  controls,  structured  dialogues  via  keyboards  or  voice, 
or  natural  language  via  keyboards  or  voice.  Implicit  communication  can  be 
accomplished  using  unobtrusive  but  direct  observations,  indirect 
measurements,  or  inference"  ( 1 9: 1 7). 

"From  a  manager's  point  of  view,  the  system  should  appear  to  be  a  helpful 
staff  assistant"  (10:27).  The  method  of  communication  between  the 
manager  and  the  computer  will  determine  if  the  system  is  a  helpful  staff 
assistant  or  a  bothersome  tool  that  is  used  only  as  a  last  resort.  There  are 
two  popular  methods  of  communication  used  on  the  microcomputers 
currently  on  the  market.  IBM*,  and  IBM  compatible,  microcomputers  provide 
a  keyboard  entry,  menu  driven  communication  interface  which  uses 
structured  dialogue.  The  Apple  Macintosh  microcomputer  provides  a  mouse 
entry,  Icon  (pictorial)  Interface,  which  is  visually  oriented  and  uses  a  single 
button  roll -around  mouse,  about  the  size  and  shape  of  a  pack  of  cigarettes, 
for  control.  Ease  of  use  and  visual  impact  are  better  on  the  mouse-icon 
microcomputer  Instead  of  typing  confusing  command  sequences  to  control 
the  computer  and  applications  software,  the  Macintosh  provides  a  "desk  top 

*  IBM  is  a  registered  trademark  of  International  Business  Machines  Corp. 


analogy,  with  documents  and  folders  and  even  a  trash  can  for  throwing 
documents  away,  that  Is  controlled  almost  exclusively  via  the  mouse. 
Therefore,  the  Macintosh  Interface  supports  those  managers  that  are  not 
typists  and  may  even  be  Intimidated  by  the  computer;  It  provides  the  user 
Interface  that  userfhendly  criteria  Indicate  necessary  for  a  manager's  DSS. 

Alter  agrees  that  the  common  belief  Is  that  managers  should  be 
encouraged  to  use  DSSs  but  he  has  found  that  problems  occur  when  untrained 
people  use  a  system  they  are  unfamiliar  with.  He  concludes  "that  the  direct 
use  of  decision  support  systems  by  nonexperts  should  be  discouraged 
rather  than  encouraged"  which  Is  "In  contrast  to  suggestions  that  better 
man-machine  Interfaces  are  the  key  to  greater  managerial  usage  of  decision 
support  systems’  (2:1 1 1-1 12).  However,  managerial  and  military  use  of 
DSSs  is  already  a  fact;  the  military  uses  many  custom  DSSs  which  were 
designed  and  written  for  a  single  application,  almost  all  of  which  were 
acquired  through  software  development  efforts  as  described  In  Chapter  I. 
The  use  of  commercial  DSS  software  will  undoubtedly  increase  as  software, 
and  microcomputer  power  and  utility  Increase. 

This  research  Is  focused  on  the  location  and  acquisition  of  a  commercial 
microcomputer  DSS  which  a  manager  can  use  for  project  selection.  As  just 
discussed,  the  Macintosh  microcomputer  provides  a  user  interface,  with 
both  the  hardware  subsystem  and  the  operation  soft  wore  subsystem,  that 
facilitates  managerial  use.  Therefore,  these  are  the  first  parts  of  the 
system  that  Is  used  for  this  research  and  only  the  application  software 
subsystem,  the  "decision  support"  in  DSS,  Is  yet  to  be  determined.  The  next 
step  in  this  location  process  Is  a  look  at  a  few  of  the  custom  DSS  software 
subsystems  that  have  been  Implemented. 


Some  Existing  DSS  Software.  The  Army  and  the  Navy  both  have  developed 
specialized  DSSs.  One  of  the  Army’s  DSSs  is  called  a  “Competition 
Decision-Assist  Package  (CDAP)"  and  had  as  a  study  objective  the 
"Development  of  a  decision-making  package  for  use  by  the  project  manager 
to  aid  making  competition  decisions’  (6).  This  DSS  requires  a 
mini-computer  (which  is  in  between  the  size  and  power  of  a  microcomputer 
and  mainframe  computer)  for  operation.  The  Navy  has  developed  a  DSS 
called  'A  Personalized  and  Prescriptive  Decision  Aid"  (6).  The  Navy  DSS  is 
implemented  on  an  IBM  personal  computer  and  was  customized  as  an  attack 
submarine  commander  s  DSS.  However,  it  Is  designed  as  a  generic  DSS.  The 
report  summary  describes  the  DSS  as  follows: 


This  report  describes  the  development  of  a  computer- 
based  display  and  analysis  system  which  caters  to  the 
personal  decision-making  styles  of  users  while  hedging 
them  about  with  safeguards  against  potential  errors 
or  biases.  The  general  conceptual  design  brings  together 
descriptive  research  in  cognitive  psychology  on  individual 
strategies  in  judgment  ana  choice,  and  prescriptive 
theories  whicn  constrain  optimal  solutions  while 
accommodating  differences  in  judgment  and  ways 
of  structuring  the  problem.  A  demonstration 
prototype  aid;  incorporating  an  advanced  user 
interface,  has  been  designed  and  partially 
Implemented  in  a  specific  testbed  and  has 
successfully  undergone  a  preliminary  test  with 
representative  potential  users.  [5:v] 


In  another  attempt  at  using  DSS  techniques,  two  Air  Force  Institute  of 
Technology  graduate  students  have  written  "A  Decision  Support  System  For 
Bare-Base  Planners"  which  addresses  the  need  for  a  specific  DSS  at  Air 
Force  Major  Command  level  (21 ).  It  is  useful  only  in  the  Intended  scenairo 
of  bare-base  planning.  In  a  different  effort  at  solving  a  specific  problem,  a 
DSS  model  was  designed  and  Implemented  to  assist  the  Xerox  corporate 


staff  to  decide  If  they  should  build  a  new  plant.  The  report  makes  no 
mention  of  what  computer  system  Is  used.  It  Is  a  good  example  of 
successful  application  of  DSS  theory  but  also  only  supports  the  scenario  it 
was  designed  to  address  (20).  One  Air  Force  custom  DSS  is  the  "Integrated 
Decision  Support  System  (IDSS)  ( 1 4)".  It  is  designed  to  aid  in  the  solution 
of  various  manufacturing  problems.  Some  of  the  many  functions  the  IDSS 
provides  is  computer  simulation  of  manufacturing  problems  to  generate 
alternative  solutions  that  are  then  evaluated  using  techniques  such  as 
financial,  statistical,  and  inventory  analysis  (14:1-2).  It  too  is 
implemented  on  a  mini -computer.  All  but  one  of  these  DSSs  is  the  result  of 
a  software  development  contract  as  described  in  Chapter  I,  and  the  one 
exception  was  the  result  of  development  efforts  by  two  Air  Force  officers. 
Next  is  a  look  at  some  of  the  commercially  available  microcomputer  DSS 
software. 

Commercial  Microcomputer  DSS  Software.  According  to  product  reviews 
and  advertisements  In  some  of  the  computing  literature,  at  least  19  DSSs  or 
DSS-type  programs  have  been  developed  that  run  on  the  two  most  popular 
microcomputers,  IBM  and  Macintosh.  These  are  listed  in  Table  I  along  with 
their  manufacturers  (in  the  Name/Mfr  column),  the  microcomputer  that  they 
operate  on,  either  IBM  or  Macintosh  (MAC)  or  both  (H/W  SYS  column),  and 
their  retail  cost  (Cost  column)  as  of  April  1986. 

As  was  mentioned  r  viously,  a  properly  used  spreadsheet  program  can 
provide  many  DSS-type  benefits.  A  decision  model  of  almost  any  complexity 
can  be  built  with  a  spreadsheet  but  It  can  be  very  hard  to  see  the 
relationships  that  are  modeled  since  they  are  not  represented  graphically 
and  nearly  impossible  for  anyone  but  the  model  designer  to  change  the 


TABLE  I. 

Macintosh/IBM  Microcomputer  D55  Software 


ft 


Name/Mfr* 

H/W  Sys 

Cost ($) 

1. 

Llghtyear/Llghtyear ,  Inc. 

IBM 

595 

2. 

Jazz/Lotus  Development  Corp. 

MAC 

595 

3. 

Lotus  1-2-3/Lotus  Development  Corp. 

IBM 

495 

4 

Trlgger/Thoughtware,  Inc. 

IBM 

495 

5. 

Reflex/Analytlca  Corporation 

IBM 

495 

6. 

Javelin/ Javelin  Software,  Inc. 

IBM 

695 

7. 

DecIslonMap/SoftStyle,  Inc. 

MAC 

145 

8 

MtndSight/Execucom  Systems  Corp. 

MAC 

195 

9. 

ods/Consultant/ODS,  Inc. 

MAC 

200 

10. 

Expert  Edge/Human  Edge  Software  Corp. 

IBM/MAC 

795 

11. 

Expert  Ease/  * 

IBM/MAC 

695 

12. 

The  Management  Edge/  " 

IBM/MAC 

250 

13. 

The  Negotiation  Edge/  ’ 

IBM/MAC 

250 

14 

The  Sales  Edge/  ’ 

IBM/MAC 

250 

15. 

The  Communication  Edge/  ' 

IBM/MAC 

250 

16. 

ExperL1sp&0PS5/Expertelligence,  Inc. 

MAC 

398 

17. 

Ensemble/  Hayden  Software  Inc. 

MAC 

300 

18. 

Excell/Microsoft  Corp. 

MAC 

395 

19. 

Quartet/Haba  Systems  Inc. 

MAC 

200 

*  Ughtyear  Is  a  registered  trademark  of  Llghtyear,  Inc. 
Jazz  and  Lotus  1  -2-3  are  registered  trademarks  of  Lotus 
Development  Corp. 
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weights  of  the  decision  factors.  Thus  they  allow  DEM  building,  although  not 
always  easily,  and  sensitivity  analysis,  which  also  is  not  easy.  This  lack  of 
ease  of  use,  or  userhostility,  means  that  few  managers  will  ever  take  the 
time  to  perform  an  analysis  of  any  complexity  using  a  spreadsheet. 

However,  if  the  results  are  properly  depicted,  perhaps  graphically,  they  can 
be  as  beneficial  to  decision  making  as  those  of  a  DSS.  Therefore,  the  new 
’integrated"  programs  such  as  Lotus  1  -2-3,  Jazz,  Quartet,  Reflex,  et  al.  that 
provide  graphics  capability  as  well  as  data  base  and/or  spreadsheet 
capabilities  fall  into  the  category  of  "decision  support"  and  are  also  listed 
for  consideration.  The  specific  criteria  for  selecting  the  decision  support 
software  are  developed  next. 

DSS  Criteria 

The  research  objective  is  to  locate,  acquire,  demonstrate  realistically, 
and  assess  a  DSS  that  aids  the  R&D  project  manager’s  project  selection 
decision  process.  The  DSS  should  aid  the  transition  from  a  semi -structured 
to  a  structured  decision  process.  The  list  of  DSS  software  has  already  been 
limited  to  those  that  are  commercially  available  and  operate  on  IBM  or 


Trigger  is  a  registered  trademark  of  Thoughtware,  Inc. 

Reflex  is  a  registered  trademark  of  Analyttca  Corp. 

Javelin  is  a  registered  trademark  of  Javelin  Sostware,  Inc. 

DecisionMap  is  a  registered  trademark  of  Softstyle,  Inc. 

MindSight  is  a  registered  trademark  of  Execucom  Systems, Inc. 
ods/Consultant  is  a  registered  trademark  of  Organizational  Development 
Software,  Inc. 

Expert  Edge,  Expert  Ease,  The  Management  Edge,  The  Negotiaton  Edge,  The 
Sales  Edge,  and  the  Communication  Edge  are  all  registered  trademarks  of 
Human  Edge  Software  Corp. 

Expertisp  and  OPS5  are  registered  trademarks  of  Expertelligence,  Inc. 
Ensemble  is  a  registered  trademark  of  Hayden  Software,  Inc. 

Excell  is  a  registered  trademark  of  Microsoft  Corp. 

Quartet  is  a  registered  trademark  of  Haba  Systems,  Inc. 
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Macintosh  microcomputers.  To  further  qualify  the  DSS  that  best  meets  the 
research  objective,  and  research  constraints  identified  in  Chapter  II,  the 
following  categories  of  DSS  performance  are  derived  from  the  literature  and 
Chapter  II,  and  further  refined  into  the  criteria  listed  below  each. 

Perform  the  Decision  Analysis.  The  decision  analysis  is  to  choose  among 
or  rank  order  several  pre-determined  alternatives.  The  first  two  criteria 
are: 


1.  The  DSS  should  allow  establishment  of  criteria/decision  factors 
for  each  specific  decision. 

2.  The  DSS  should  in  some  way  rank-order  pre-determined  projects 
based  on  the  decision  factors. 

Support,  Not  Replace,  the  Manager's  Judgment.  The  manager's  judgment 
is  necessary  to  determine  the  importance  of  decision  factors  in  the  DEM,  and 
may  be  required  to  determine  a  qualitative  measure  of  merit  for  an 
alternative.  The  next  two  criteria  are: 

3.  The  DSS  should  allow  different  weights/priorities  to  be  assigned 
to  each  different  decision  factor  in  the  DEM. 

4.  The  DSS  should  allow  the  ranking  of  each  project  alternative  on 
the  decision  factor. 

Structure  the  Decision  Environment.  Prince  found  that  the  decision 
techniques  most  often  used  by  project  managers  are  checklists  and  profile 
charts,  scoring  models,  decision  trees,  and  goal  programming  (15:36). 
Therefore,  a  project  manager's  DSS  should  aid  the  manager  in  accomplishing 
one  of  these  techniques.  Thus,  the  single  criteria  for  this  category  is: 

5.  For  the  R&D  environment,  the  DSS  should  aid  at  least  one  of  the 
project  selection  techniques  such  as  checklists  and  profile  charts,  scoring 
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models,  decision  trees  or  goal  programming. 

Userfriendly.  Both  the  hardware  and  software  must  work  together  to 
provide  a  userfriendly  managerial  system.  Two  criteria  for  judging  this  are. 

6.  The  DS5  should  allow  easy  DEM  building. 

7.  The  D55  should  facilitate  sensittvity  analysis  after  the  DEM  is 
built  and  the  data  entered. 

Provide  a  Relevant  Output.  The  output  should  be  easy  to  understand  and 
should  not  require  searching  through  extraneous  material  to  find  the  result. 
To  provide  this: 

8.  The  DS5  should  provide  multiple  graphic  and  text  report  formats. 

9.  The  DSS  should  allow  comparison/analysis  of  the  results  with 
respect  to  the  decision  criteria/ inputs. 

Additional  Research  Criteria.  The  criteria  (constraints)  identified  in 
Chapter  II  are: 

1 0.  The  DSS  must  operate  on  a  Macintosh  microcomputer. 

1 1 .  The  DSS  can  be  acquired  by  3 1  April  1 986. 

1 2.  The  DSS  can  be  bought  for  under  $  1 00. 

This  list  of  12  criteria  Is  the  yardstick  by  which  all  of  the  candidate  DSSs 
are  measured.  The  next  section  provides  a  matrix  chart  for  the  selection  of 
the  DSS  software  that  best  satisfies  these  criteria. 

DSS  Software  Selection 

The  19  DSS  software  programs  identified  previously  can  now  be  rated 
against  the  1 2  OSS  criteria  listed  above.  The  DSS  /Criteria  Matrix  is  shown 
below  in  Table  II.  The  left  column  of  numbers  corresponds  to  each  DSS  as  it 


TABLE  II. 


DS5  Software/Criteria  Matrix 

Criteria «.  1  23456789  10  11  12  Total  Y/N 

DSS:  1  YYYYYYYYYN  Y  N  10/2 

2  YYYYNNNYYY  Y  N  8/5 

3  YYYYNNNYYN  Y  N  7/6 

4  YYYYNNNYYN  YN  7/6 

5  YYYYNNNYYN  YN  7/6 

6  YYYYYYYYYN  YN  10/2 

7  YYYYYYYYYY  Y  N  11/1 

8  YYYYYYYYYY  YN  11/1 

9  YYYYNYYYYY  Y  N  10/2 

10  YYYYNNYYYY  Y  N  9/3 

11  YYYYNNYYYY  YN  9/3 

12  NNNNNYNNYY  Y  N  4/8 

13  NNNNNYNNYY  YN  4/8 

14  NNNNNYNNYY  YN  4/8 

15  NNNNNYNNYY  YN  4/8 

16  YYYYNNYYYY  YN  9/3 

17  YYYNNNNYYY  Y  N  7/5 

18  YYYNNNNYYY  YN  7/5 

19  YYYNNNNYYY  YN  7/5 


Is  listed  in  the  section  on  commercial  microcomputer  D55s.  The  top  row  of 
numbers  corresponds  to  each  of  the  12  DSS  criteria  derived  previously.  In 
the  table,  Y  ■  yes  and  N  *  no;  each  DSS  Is  scored  against  each  of  the  criteria 
according  to  whether  It  meets  the  criteria  (Y)  or  does  not  satisfactorily 
meet  the  criteria  (N).  The  last  column  under  Total  Y/N  Is  the  total  number 
of  yes  answers  and  no  answers  for  each  DSS.  For  example,  the  last  DSS 


listed  (19)  has  seven  yes  answers  and  five  no  answers  which  results  in  an 
entry  of  7/5  in  the  Total  Y/N  column.  None  of  the  DSSs  listed  satisfy  all  of 
the  criteria,  but  DS5  numbers  7  and  8  both  have  1 1  yes  answers.  The  no 
answer  for  both  is  a  result  of  the  price  criteria.  $  145  for  number  7  and 
$  1 95  for  number  8.  Since  both  DSS  numbers  7  and  8  have  the  same  criteria 
score  and  operate  on  a  Macintosh  microcomputer,  DSS  number  8  is 
eliminated  due  to  the  higher  price.  In  fact,  when  DSS  number  7, 
Decisionhap,  was  ordered  the  price  had  dropped  to  $76  while  DSS  number  8 
was  still  over  $100.  Therefore,  DSS  number  7  is  the  only  one  that  actually 
meets  all  12  criteria. 

Thus,  the  first  part  of  the  research  objective,  to  locate  and  acquire  a 
DSS,  is  complete;  the  hardware  subsystem  is  a  Macintosh,  which  includes 
the  operation  software  subsystem,  and  the  application  software  subsystem 
is  DecIsionMap.  The  next  part  of  the  research  objective  is  to  realistically 
demonstrate  and  assess  the  DSS  to  see  If  It  aids  the  project  selection 
decision  process.  The  data  necessary  to  enable  a  realistic  assessment  is 
discussed  next. 

The  Data 

Inherent  In  performing  a  realistic  assessment  of  this  DSS  is  the  data. 
The  use  of  simulated  data  would  provide  only  a  simulated  utility 
assessment.  Therefore,  actual  project  data,  which  are  derived  from  the 
respective  projects,  are  the  best  test  of  DSS  utility.  The  data  for  this 
assessment  were  derived  through  a  complicated  methodology  that  involved 
numeric  scoring  and  written  comments  on  each  of  49  projects  and  resulted 
in  an  unusually  large  amount  and  high  quality  of  data  not  normally  available 


to  the  project  manager-decision  maker  upon  which  to  base  a  project 
selection  decision.  In  brief  summary,  the  method  chosen  for  this  task  was 
first  to  provide  a  general  overview  of  specific  Al  research  areas  and 
example  Al  projects  to  various  experts  in  industry,  academia,  and  the 
government.  Each  expert  was  requested  to  suggested  at  least  one  Al 
research  project  of  his  or  her  own  design.  The  experts  were  then  assembled 
to  judge  the  merits  of  the  prospective  research  projects.  Thus,  the  research 
planning  workshop  for  artificial  intelligence  in  manufacturing  was  held  on 
31  July  1985  to  1  August  1985  in  which  the  experts  were  each  assigned  to 
one  of  three  groups  corresponding  to  the  project  application  areas  of  unit 
processes  (Group  1 ),  manufacturing  systems  (Group  2),  and  intelligent 
information  handling  (Group  3).  Each  panel  of  experts  then  scored  the 
suggested  projects  in  their  area  on  criteria  such  as  impact,  payoff,  and 
technical  feasibility.  The  Group  I  projects  were  scored  on  a  total  of  five 
criteria  while  the  Groups  2  and  3  projects  each  were  scored  on  six  criteria. 
Each  criteria  was  scored  on  a  nine  point  Likert  scale  where  one  indicated  a 
strong  disagreement  and  nine  a  strong  agreement.  In  addition,  the  scoring 
methodology  required  each  panel  of  experts  to  score  and  discuss  the 
projects  in  three  assessment  rounds  in  which  commentary  was  specifically 
solicited  from  those  experts  that  had  marked  a  project  significantly  higher 
or  lower  than  the  mean  score,  also  called  the  outlier  comments,  in  an  effort 
to  capture  information  not  apparent  with  numeric  scores  alone.  ( 12: 1  -7) 

In  total,  the  Al  project  data  presents  the  manager,  faced  with  selecting 
which  projects  to  pursue,  a  variety  of  information  with  which  to  make  the 
choice;  the  project  data  consists  of  numeric  data  which  gives  a  quantitative 
value  for  each  project  and  expert  (outlier)  commentary  data  which  gives  a 


qualitative  value  provided  by  those  experts  that  rated  a  project 
significantly  different  from  the  mean  score.  Thus,  a  D55  could  aid  the 
project  selection  process  by  assisting  the  transition  from  a 
semi -structured  to  a  structured  decision  process.  This  assistance  could  be 
provided  through  an  evaluation  of  the  numeric  data  along  with  an  indication 
of  the  outlier  comments  that  are  associated  with  each  project  such  that  the 
project  manager  is  provided  a  comparative  view  of  each  project  with 
respect  to  the  other  projects  in  a  group. 

The  whole  system  can  be  exercised  to  determine  its  utility  once  the  data 
are  entered  in  the  DSS.  The  majority  of  the  utility  evaluation  is  as  a  result 
of  the  data  preparation,  data  entry,  and  resultant  DEM  manipulation  and 
decision  results  that  are  necessary  for  this  research  project.  However, 
since  this  DSS  is  for  managerial  use  the  satisfaction  of  the  intended  users, 
managers,  is  paramount.  Therefore,  the  instrument  used  for  determining 
manager/user  satisfaction  with  respect  to  this  DSS  is  discussed  in  the  next 
section. 


The  Management  Questionnaire 

The  last  section  in  this  chapter  is  on  the  management  questionnaire  that 
is  used  to  assess  the  degree  of  satisfaction  that  managers  associate  with 
this  DSS.  "Satisfaction  of  users  with  their  information  systems  is  a 
potentially  measurable,  and  generally  acceptable,  surrogate  for  utility  in 
decision  making'  (8:785)  and  several  studies  have  been  conducted  on  how  to 
measure  user  satisfaction  (1  and  8).  The  instruments  that  have  been  used 
for  measuring  user  satisfaction  contain  questions  that  cover  the  range  from 
DSS  development  and  vendor  support  to  DSS  utility  as  perceived  by  the  user 


(1:336, 8:798).  Since  many  of  these  areas  are  irrelevant  to  this  effort  a 
questionnaire  was  developed  (Table  III)  that  incorporates  those  areas  that 
are  relevant.  In  addition,  since  the  DSS  criteria  for  this  project  were 
developed,  addressed,  and  met  on  a  case  by  case  basis  previously  and  the 
‘best  fit'  was  chosen,  the  questions  are  designed  to  assess  the  utility  of 
this  DSS  as  it  is  perceived  by  the  manager/user. 

The  general  assessment  areas  are  the  DSS  capabilities,  output, 
preparation,  userfriendliness,  and  overall  satisfaction  with  the  DSS.  These 
areas  correspond  to  question  groups  1.2,  2.0  and  3.0  developed  by  Adelman, 
Rook,  and  Lehner  ( 1 :336).  The  specific  questions  on  the  management 
questionnaire  relate  to  specific  points  and  perceptions  to  assess  about  this 
DSS  and  also  correspond  to  specific  points  of  interest  as  indicated  by 
Adelman  et  al.,  and  Reimann  and  Waren  ( 1 :336,  1 8. 1 68).  Each  question  is 
scored  on  a  five  point  Likert  scale,  as  indicated  in  Table  III,  to  indicate  the 
managers'  degree  of  agreement  with  the  question.  Thus,  questions  1  through 
4  ask  the  managers'  perceptions  about  how  well  the  DSS  fits  his  or  her 
decision  environment.  The  perceived  benefit  and  speed  of  the  DSS  output  is 
judged  by  questions  5  through  7  end  the  manager's  judgment  of  ease  of  DSS 
preparation  for  future  decision  situations  is  asked  by  questions  8  and  9. 
Next,  the  userfriendliness  of  the  DSS  as  a  whole,  and  the  hardware 
subsystem  and  software  subsystem  individually,  Is  asked  by  questions  10 
through  12.  Finally,  the  manager's  actual  perception  of  overall  satisfaction 
with  the  system  is  addressed:  Would  Improved  decision  making  result  if  he 
or  she  used  this  DSS,  would  the  manager  use  it,  and  recommend  Its  use? 
These  are  asked  by  questions  13  through  15.  Comments  and 


TABLE  III. 

MANAGEMENT  QUESTIONNAIRE 


Use  a  number  from  the  scale  below  to  indicate  your 
agreement/disagreement  with  questions  1-15. 

NO  YES 

I - 1 - 1 - 1 - 1 

1  2  3  4  5 

I.  Is  sufficient  data  incorporated  for  this  to  be  a  useful  decision  aid? 


2.  Could  sufficient  data  be  incorporated  for  future  decisions? 


3.  Does  this  decision  aid  react  reasonably  when  manipulated? 


4.  Does  this  decision  aid  simulate  real  world  relationships? 


5.  Does  this  decision  aid  provide  an  understandable  output/result? 


6.  Is  the  output/result  relevant  to  the  decision? 


7.  Is  the  decision  aid  operation  sufficiently  fast? 


8.  Do  you  think  that  future  decision  aid  preparation  would  be  easy? 


9.  Do  you  think  that  future  data  preparation  would  be  easy? 


1 0.  Is  the  system  (computer  and  software)  easy  to  use? 


11.  Is  the  computer  easy  to  use? 


1 2.  Is  the  software  easy  to  use? 


1 3.  Would  this  decision  aid  Improve  your  decision  making? 


1 4.  Would  you  use  this  system? 


1 5.  Would  you  recommend  use  of  this  system? 


1 6.  Comments  or  recommendations? 


recommendations  are  solicited  for  any  area  of  interest  or  perception  that 
the  manager  wants  to  add. 

Conclusion 

Chapter  I  set  the  stage  for  the  potential  use  of  commercial  software  for 
Air  Force  applications  and  identified  the  specific  application  as  a 
managerial  decision  support  system  for  project  selection.  In  this  chapter 
the  system,  Macintosh  hardware  and  operation  software  subsystems  and 
DecisionMap  application  software  subsystem,  has  been  selected.  The 
project  data  necessary  for  a  realistic  demonstration  of  this  DS5  has  been 
discussed  and  the  instrument  for  assessing  managerial  perceptions  about 
this  DSS  has  been  developed.  The  next  step,  operating  the  DSS  with  the  data 
is  performed  in  Chapter  IV;  the  results  of  exercising  the  system  with  the 
data  and  the  managerial  assessment  are  discussed  in  Chapter  V. 


Introduction 

Chapter  III  concluded  with  the  selection  of  the  DSS  program  DecisionMap 
for  the  data  analysis  and  managerial  use,  and  development  of  the 
questionnaire  for  subsequent  managerial  feedback.  In  this  chapter  the 
general  utility.  Including  the  decision  model, -the  date,  and  the  operating 
steps  for  this  DSS,  Is  discussed  first.  Then  the  specific  utility  and 
operation  with  respect  to  the  Al  project  DEhs  and  project  dota  ore 
discussed,  followed  with  the  DSS  results  for  each  of  the  three  groups  of 
projects.  One  method  for  including  the  expert  commentary,  collected  when 
the  projects  were  scored,  into  the  decision  process  is  discussed  and  last,  an 
example  sensitivity  analysis  is  performed.  The  managerial  assessment  of 
this  DSS,  as  well  os  the  Impact  of  the  general  and  specific  DSS  utility  on 
the  prospective  manager/user  is  discussed  In  Chapter  V 

General  Utility:  The  Decision  Model  and  Data  Structure 

Previous  research  by  Prince  identified  one  of  the  techniques  preferred  by 
managers  for  project  selection  as  a  decision  tree  (15).  The  format  used  by 
DecisionMap  is  a  decision  tree,  it  provides  a  structured  tree  format  for 
graphically  building  decision  models.  Each  branch  of  the  tree  has  a  decision 
foctor,  or  criterion,  at  the  end  of  it.  The  criteria  can  each  be  weighted,  to 
indicate  their  importance  in  the  model,  either  as  a  quantitative  amount  on  a 
scale  from  1  to  100  or  as  a  qualitative  amount  on  a  less-to-more  weight 
scale.  They  con  be  called  ony thing  from  "Bosses'  Preference"  to  "Cl",  within 
a  20  character  limit.  After  the  decision  criteria  are  identified,  each 


alternative  is  ranked  on  the  criteria,  again  on  either  a  quantitative  or 
qualitative  scale.  Then  the  decision  results  are  displayed  as  text  or  graphic 
outputs  and  the  decision  factor  weights  can  be  shifted  for  a  sensitivity 
analysis.  Thus,  Decisionflap  provides  a  structured  decision  tree  format  that 
fits  any  decision  situation  that  requires  a  choice  among  pre-determined 
alternatives,  where  the  relevant  criteria  for  the  decision  can  be  identified, 
and  the  alternatives  con  be  ranked  on  the  criteria  either  quantatively  or 
qualatively. 

Operating  Steps.  The  steps  that  must  be  followed  to  operate  this  DSS 
fall  into  three  categories:  DEM  building,  data  entry,  and  sensitivity  analysis. 
The  steps  should  be  followed  in  order  if  a  new  decision  onalysis  is  initiated, 
but  the  decision  analysis  could  begin  with  data  entry  if  a  template  is  used  , 
or  sensitivity  analysis  if  new  data  (or  new  weights  for  the  decision  factors) 
for  a  pre-built  DEM  is  used.  Throughout  the  following  steps  "menu 
selections’  refer  to  the  series  of  words  in  the  menu  bar  across  the  top  of 
the  computer  screen,  each  of  which  may  be  selected  with  the  mouse  causing 
a  "pop  down  menu"  to  oppear  with  DSS  options  in  it.  Various  actions, 
indicated  by  the  menu  options,  con  then  be  caused  to  occur  by  selecting  one 
of  the  menu  items.  The  complete  sequence  begins  with  turning  on  the 
computer,  inserting  the  Decislonhap  disk  and  continues  as  follows: 

1.  Select  and  open  the  DecisionMap  icon  by  "pointing  and  clicking,"  which 
means  to  move  the  mouse  so  the  pointer/arrow  is  over  the  desired  icon  and 
press  the  mouse  button  twice.  This  process  starts  the  DSS  software. 

2.  The  software  automatically  requests  a  name  for  a  new  decision, 
which  is  entered  via  the  keyboard.  Any  name  up  to  20  characters  long  may 
be  used,  after  which  "OK"  is  entered  by  the  mouse  os  explained  above. 
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3.  A  blank  DEM  appears  with  the  decision  name  at  the  'root'  of  the 
decision  tree.  The  decision  factors  are  then  entered  by  selecting,  again 
with  the  mouse,  an  empty  factor  rectangle.  The  software  requests  a  name, 
not  more  then  20  characters  long,  for  the  decision  factor  which  is  entered 
by  the  keyboard.  When  OK  is  selected  by  the  mouse  the  factor  is  added  to 
the  decision  tree.  Thus,  a  decision  tree  is  built  by  adding  all  the  decision 
factors;  a  factor  on  Level  1,  the  first  layer  of  decision  factors,  can  be 
sub-divided  into  five  more  factors  on  Level  2  and  each  of  these  can  be 
further  sub-divided  in  five  more  factors  on  Level  3.  Therefore,  there  con  be 
a  total  of  five  factors  on  Level  1, 25  on  Level  2,  125  on  Level  3  and  and  so 
on  until  the  maximum  memory  of  the  computer  is  reached,  in  addition,  if  a 
decision  required  25  factors,  the  Level  1  factors  could  be  used  as  ’dummy" 
factors,  or  non-weighted  place  holders,  for  the  25  factors  in  Level  2. 

4.  After  the  decision  tree  is  complete  the  factors  are  weighted  as  to 
their  importance  in  the  decision  model.  This  is  accomplished  first  by 
selecting  the  level  above  the  factors  to  be  weighted  and  from  which  the 
factors  are  sub-divided,  called  the  summary  factor.  For  example,  the 
Level  1  factors  are  weighted  from  the  root  decision  name,  selected  by  the 
mouse,  and  then  selecting  the  ’Weight"  option  from  the  "Factors’  menu.  The 
weight  screen  presents  the  user  a  set  of  columns  with  the  name  of  the 
factors  under  them.  Each  column  height  is  adjusted,  with  the  mouse,  to  a 
height  corresponding  to  the  importance  of  the  factor  in  the  decision.  The 
weight  may  be  judged  qualatively  on  a  less  to  more  scale  or  quantatively  on 
a  1  to  100  scale.  The  columns  are  the  same  for  both  scales  but  the  numeric 


values  are  displayed  in  the  columns  when  the  ’Numeric’  option  is  selected 
from  the  ’Decision*  menu.  After  all  of  the  factors  are  weighted,  the  DEM  is 
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complete  ond  ready  for  the  alternatives  to  be  added. 

5.  Selecting  the  "List’  option  from  the  ‘Alternatives"  menu  presents  the 
user  with  a  screen  for  listing  the  alternatives.  They  are  listed  by  selecting 
a  blank  alternative  rectangle,  typing  in  a  name  of  20  characters  or  less  and 
comments  of  50  characters  or  less,  and  selecting  OK  to  continue.  A 
maximum  of  five  alternatives  can  be  listed  in  each  decision  tree,  which  are 
then  ranked  on  the  last  decision  factors  at  the  end  of  each  branch  of  the 
decision  tree,  called  the  detail  factors. 

6.  The  alternatives  are  ranked  on  the  detail  factors  by  first  selecting 
one  of  the  detail  factors  and  then  the  ‘Rank*  option  from  the  “Alternatives" 
menu.  The  rank  screen  is  similar  to  that  of  the  weight  screen  except  that 
the  columns  have  the  alternatives  listed  under  them  instead  of  the  decision 
factors.  Each  alternative  is  ranked  on  the  decision  factor,  either 
qualotively  on  a  worst  to  best  scale  or  quantatfvely  on  a  1  to  100  scole,  as 
described  in  Step  4.  When  the  alternatives  are  each  ranked  on  all  of  the 
detail  factors  the  decision  results  can  be  viewed  and  a  sensitivity  analysis 
performed. 

7.  Selecting  the  “Results"  option  from  the  "Decision"  menu  presents  the 
user  with  either  the  rank-ordered  decision  results  or  additional  analysis 
outputs,  some  of  which  will  be  discussed  later.  In  addition,  to  perform  a 
sensitivity  analysis,  the  user  can  switch  between  the  decision  results 
output  and  the  weight  screen  to  change  the  weights  of  the  decision  factors 
ond  then  switch  back  to  view  the  effects  of  the  change  on  the  results. 

The  seven  steps  discussed  above  are  the  basic  process  for  operating  this 
DSS.  The  DSS  user  con  devise  additional  analysis  techniques  of  his  or  her 
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own;  decision  factors  con  be  added  or  deleted  to  see  the  results  on  the 
decision  in  addition  to  adding  pertinent  decision  information  in  the 
comments  of  each  alternative,  in  fact,  the  latter  technique  is  used  later 
with  the  Al  project  data  as  on  indicator  of  relevant  decision  information 
that  otherwise  might  not  be  provided  to  the  decision  maker.  These  seven 
steps,  along  with  relevant  comment  information,  are  the  process  used  for 
the  specific  utility  analysis  which  includes  building  the  project  DEhs, 
entering  the  project  data,  and  an  example  sensitivity  analysis. 

Specific  DSS  Utility:  The  Project  Data 

The  numeric  doto  set  for  each  of  the  three  project  groups  was  collected 
during  the  Reseorch  Planning  Workshop  for  Artificial  Intelligence  in 
Manufacturing.  As  discussed  previously,  the  projects  were  scored  by 
experts  in  the  areas  of  unit  processes,  manufacturing  systems,  and 
Intelligent  information  handling  for  Project  Groups  1, 2,  and  3  respectively. 
The  Group  1  score  sheet  consists  of  five  criteria  (Cl  through  C5)  while  the 
Group  2  and  3  score  sheets  consist  of  six  criteria  (Cl  through  C6)  each. 

Each  criteria  is  ranked  on  a  nine  point  Likert  scale.  Group  1  contains  15 
projects.  Group  2  contains  20  projects,  end  Group  3  contains  14  projects. 
Therefore,  there  is  a  maximum  of  1,080  possible  numbers  necessary  to  fully 
describe  the  Group  2  scoring  data,  (20  projects  X  6  criteria  X  9  point 
scale  =  1,080),  756  for  Group  3,  and  675  for  Group  1. 

DEM  Data.  Due  to  the  large  number  of  possible  inputs,  the  meon  score  for 
each  criteria  is  used  as  the  basis  for  the  numeric  input  doto  in  the  DEMs  for 
the  decision  analysis.  This  requires  a  great  deal  of  extra  data  preparation, 
in  computing  the  meon  of  each  score  and  then  entering  them  into  the 


decision  model,  ond  results  in  o  lock  of  versatility  in  identifying  those 
projects  thot  moy  hove  o  bi-modol,  or  split,  distribution  due  to  o  wide  range 
of  scores.  Such  projects  moy  be  of  further  interest  to  the  decision  moker  if 
the  accompanying  outlier  commentary  indicates  a  discrepancy  with  the  DSS 
numeric  results;  a  technique  is  needed  to  link  the  DSS  numeric  results  to 
the  expert  commentary  since  the  row  data  is  not  present  in  the  DSS  to 
provide  the  link.  The  DecisionMop  software  is  not  modifiable  nor  does  it 


provide  access  to  other  software  to  supplement  its  capabilities.  One 
method  for  providing  the  necessary  link  between  the  numeric  results  end  the 
expert  commentary  is  presented  next. 

Aggregrote  Standard  Deviation.  A  measure  of  the  score  dispersion  is 
indicated  by  the  standard  deviation  of  the  Likert  scale  scores.  Since  outlier 
comments  were  specifically  solicited  from  those  experts  that  ranked  a 
project  criterion  significantly  different  from  the  mean  of  the  panel, 
dispersion  can  be  related  to  the  commentary.  In  other  words,  the  experts 
that  caused  a  large  standard  deviation  also  wrote  the  comments.  The 
aggregrote  standard  deviation  (ASD)  is  simply  the  sum  of  the  individual 
stendard  deviations  of  the  five  criteria  for  that  project.  The  ASD  is 
included  in  the  DSS  text  report  (Figure  4)  under  the  comments  column,  next 
to  the  respective  project  score.  It  provides  a  quick  system  tie  to  the 
verboge. 

It  is  difficult,  at  best,  to  judge  the  variance  in  scores  on  a  project  by 
looking  at  the  individual  standard  deviations  since  they  only  very  from  just 
under  1.0  to  approximately  2.5;  differences  in  project  scoring  ere  very  herd 
to  detect  through  a  row  of  numbers.  Therefore,  the  sum  of  the  row,  the  ASD, 
is  used  as  on  indicator  of  score  variance.  A  project  that  is  numerically 
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ranked  very  high  or  low  by  the  DSS  and  has  a  comparatively  high  ASD  should 
be  carefully  reviewed  by  the  decision  maker  to  determine  why,  and  if  it 
should  be  considered  for  further  development  regardless  of  the  DSS  rank. 

The  individual  criterion  standard  deviation  (Appendix  B)  can  be  inspected  to 
determine  if  one  or  two  specific  criteria,  or  all  criteria,  contribute  to  the 
wide  score  dispersion.  In  either  case,  the  expert  commentary  (Appendix  A) 
can  be  reviewed  to  learn  the  experts'  opinions  about  the  project.  Thus,  not 
only  is  numerical  ranking  and  analysis  facilitated,  but  the  judgment  of  the 
experts  on  the  subject  con  be  colled  upon,  when  needed,  to  assist  the 
decision  maker  in  the  qualitative,  semi-structured,  decision  process. 

Data  Entry.  As  mentioned  obove,  the  mean  criterion  score  is  the  basis 
for  the  numbers  entered  into  the  DEM.  For  example,  the  Likert  scale  for 
criterion  1  of  project  1  in  group  1  is  scored: 

Scale:  123456789 


Score.  001324500 

This  results  in  a  Cl  mean  of  (3X1  ♦  4X3  ♦  5X2  ♦  6X4  ♦  7X5)  +  15  r  5.60. 
However,  DecisionMop  handles  input  data  rounded  to  the  nearest  0.5  and 
requires  a  scale  from  1  to  100.  Therefore,  to  minimize  the  effects  of 
rounding  and  to  facilitate  data  entry  on  the  1  to  100  scale,  the  means  are 
multiplied  by  10  and  a  mean  of  5.47  is  entered  as  54.5,  7.13  as  71.5, 6.21  as 
62.0, 3.68  as  37.0,  etc.  The  tables  of  meens/entered  values,  standard 
deviations,  ASDs,  and  an  examination  of  the  rounding  error,  ore  in 
Appendix  B. 


DSS  Results 

DecisionMap  can  operate  on  only  five  alternatives  at  a  time.  Therefore, 
the  project  groups  must  be  divided  into  sets  of  five  or  less.  This  results  in 
four  DEMs  for  Group  1  (which  has  15  projects),  five  for  Group  2  (20 
projects),  and  four  for  Group  3  (14  projects).  The  extra  DEM  in  each  group 
provides  for  ranking  the  top  five  projects  identified  in  each  set  of  five.  The 
general  DSS  operating  procedure  was  discussed  previously;  only  a  brief 
explanation  and  the  results  are  presented  here. 

Group  1,  Projects  1-5  Results.  After  the  DEM  is  built  and  the  project 
scores  entered  as  the  rank  on  each  criterion,  the  decision  results  for 
projects  1  through  5  can  be  viewed  as  text,  graphically,  or  in  two  different 
graphic  comparisons  for  analysis.  The  Text  Report  (Figure  4)  presents  the 
projects  ordered  from  highest  (on  top)  to  lowest  (on  bottom)  numerical  rank 
with  the  ASD  for  each  project  listed  in  the  comments.  The  Graphic/Numeric 
Results  (Figure  5)  show  the  relative  rank  of  each  alternative  compared  to 
the  other  in  a  graphic  format.  Either  Figure  4  or  5  will  show  the  decision 
maker  the  top  ranked  alternative,  but  only  Figure  4  indicates  the  ASD,  which 
is  discussed  more  later.  The  Numeric  Comparison  (Figure  6)  indicates  the 
importance  of  each  criteria  in  the  total  score  for  each  project.  Last,  the 
Graphic  Comparison  (Figure  7)  is  an  illustration  of  the  individual  score  that 
each  project  had  on  a  specific  criterion.  Each  figure  is  taken  directly  from 
the  DecisionMap  program  but  they  have  been  edited  to  better  fit  in  this 
report.  Therefore,  the  vertical  scales  are  not  full  size  but  the 
representations  are  accurate  otherwise. 
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Figure  4.  Group  1  Projects  1-5  DSSText  Report 


Figure  5.  Group  1  Projects  1-5  Graphic/Numeric  Results 
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Figure  6.  Group  1  Projects  1-5  Numeric  Comparison  of  Results 
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Figure  7.  Group  1  Projects  1-5  Graphic  Comparison  of  Results 


As  mentioned  above.  Figures  6  and  7  illustrate  one  of  the  analysis  tools 
provided  by  this  DSS.  Figure  7  shows  how  each  project  compares  with 
respect  to  the  others  on  the  Level  1  criteria,  and  can  be  used  by  the  decision 
maker  to  see  which  alternatives  scored  highest  or  lowest  on  a  particular 
criterion.  However,  Figure  6  gives  a  visual  indication  of  the  contribution 
eoch  criterion  mokes  toward  the  total  score  of  a  project.  A  sensitivity 
analysis  is  facilitated  by  changing  the  weight  of  one  of  the  criteria  and  then 
viewing  the  resultant  change  on  the  numeric  comparison.  Thus,  the 
importance,  or  insignificance,  of  a  particular  criterion  on  the  decision 
results  can  can  be  revealed  by  a  large  change  in  the  result  or  very  little 
change  in  the  result. 

Group  1,  Top  5  Projects.  The  decision  process  described  above  results  in 
identification  of  two  high  OSS  score  projects  in  the  first  five  project  set  in 
Group  1.  As  mentioned  previously,  four  OEMs  are  built  for  Group  1:  One  for 
projects  1  through  5,  one  for  projects  6  through  10,  one  for  projects  1 1 
through  15,  and  one  for  the  top  five  projects  identified  in  the  other  three 
DEMs.  The  same  DEM  is  used  for  each  set  of  five  projects  and  the  project 
data  are  entered  exactly  the  same  way.  Then  the  projects  with  the  highest 
numeric  scores  are  reentered  into  the  Top  5  DEM  to  prioritize  them.  Thus, 
the  top  five  projects  in  Group  1  are  listed  in  order  in  Figure  8  and  shown 
graphically  in  Figure  9. 


Figure  8.  DSS  Text  Results  on  Group  1  Top  5  Projects 


Figure  9.  DSS  Numeric  /Graphic  Results  on  Group  1  Top  5  Projects 


Groups  2  and  3  DEM.  Groups  2  and  3  are  treated  like  Group  1  with  the 
exception  of  adding  criteria  5  and  6  to  level  2  via  the  dummy  factor  called 
Time  &  Resources,  as  discussed  in  Step  3  of  the  operation  steps.  The  weight 
of  the  Time  and  Resources  factor  is  twice  that  of  the  other  factors  since  it 
contains  two  criteria. 

Group  2,Top  5  Projects.  The  following  figures,  10  and  1 1,  are  the 
decision  results  for  the  Group  2  projects. 
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Figure  1 1.  DSS  Numeric/Graphic  Results  Group  2  Top  5  Projects 


Group  3,  Top  5  Projects.  Lost,  the  Group  3  top  five  projects  ore 
presented  in  Figures  12  and  )3.  The  Group  3  DEM  is  the  seme  os  the  Group  2 
DEM;  the  data  preparation  and  entry  is  the  some  for  both  groups  also. 
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Figure  12.  DSS  Text  Results  on  Group  3  Top  5  Projects 

49 


OWWWWSWjW'PiwIiiMWiJWwIiWvIwiISV 


Figure  13.  DSS  Numeric/Graphic  Results  on  Group  3  Top  5  Projects 


ASP  Analysis.  As  discussed  previously,  the  ASD  is  included  to  decrease 
some  of  the  uncertainty  associated  with  the  project  ranks.  For  example,  the 
ASPs  corresponding  to  the  first  five  projects  in  Group  1  (Figure  4)  indicate 
an  increasing  score  dispersion  as  the  project  rank  decreases;  there  is  more 
agreement  by  the  experts  on  the  top  two  projects,  4  and  5  with  ASPs  of  5.91 
and  6.15  respectively,  than  there  Is  on  the  bottom  two  projects,  2  and  3 
with  ASPs  of  7.73  and  9.68  respectively.  The  ASPs  for  the  top  five  projects 
in  Group  1  (Figure  8)  however,  show  no  trend  and  little  variance  (5.77  to 
6.77).  This  may  indicate  an  agreement  by  the  experts  as  to  the  relative 
merits  of  these  projects. 

The  ASPs  for  the  top  five  projects  In  Groups  2  (Figure  10)  and  3 
(Figure  12)  are  larger  due  to  more  decision  criteria  (Six  for  Groups  2  and  3, 


five  for  Group  1).  Also,  these  indicators  have  a  wider  spread,  from  7.10  to 
10.26  for  the  Group  2  top  five  projects  and  from  6.69  to  9.40  for  the  Group  3 
top  five  projects.  This  indicates  more  disagreement  by  the  experts  on 
scoring  both  groups. 

Two  points  are  noteworthy  on  the  ASD: 

1.  Two  projects  in  the  Group  2  top  five,  projects  5  and  2,  and  one 
project  in  the  Group  3  top  5,  project  14,  have  high  ASDs.  Each  of  these 
projects,  and  their  corresponding  expert  commentary,  should  be  reviewed 
before  proceeding  with  them.  For  example.  Group  2  Project  2  has  a  total  of 
seven  comments,  three  of  which  consider  it  too  narrow  in  scope  while  two 
other  comments  suggest  either  that  this  project  is  not  on  innovative  Al 
application  or  alternative  approaches  to  the  problem.  The  Project  2 
negative  comments  suggest  that  although  the  DSS  numeric  score  is  high  the 
project  is  not  worth  pursuing.  Group  1  Project  5  has  four  comments;  three 
say  that  there  is  not  enough  knowledge  about  or  explanation  of  the  project 
and  one  says  the  impact  is  substantial  but  says  nothing  about  how  the 
project  is  to  be  accomplished.  Having  such  a  relatively  high  ASD  and 
negative  comments.  Project  5  of  Group  2  should  be  eliminated  also.  On  the 
other  hand.  Group  3  Project  14  has  only  four  comments  all  of  which  agree 
that  this  is  an  ambitious,  important,  but  very  difficult  project.  The  Project 
14  comments,  unlike  those  discussed  above,  reveal  that  the  experts  think 
this  project  is  worthwhile  and  should  be  developed  with  the  awareness  of 
Us  difficulty  and  possible  problem  areas. 

2.  At  least  one  project  in  each  group  has  a  very  high  relative  ASD  and 
was  not  ranked  in  the  top  five  projects  for  that  group  by  the  DSS. 

(Appendix  B,  Group  1  project  7,  Group  2  project  1 1,  Group  3  project  7) 
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These  projects,  end  commentory,  should  be  reviewed  to  determine  the  ceuse 
for  the  dispersion,  if  possible,  end  to  decide  if  one  or  more  of  these 
projects  should  be  considered  for  further  development  regerdless  of  the 
actual  scores  by  the  experts. 

Group  1  Project  7  has  four  comments,  three  questioning  the  project’s 
concept,  objectives,  and  technical  feasibility  and  one  suggesting  that  the 
basis  of  the  work  is  already  accomplished.  Project  7  is  rightfully  scored 
low  by  the  DSS  and  should  not  be  considered  further. 

Group  2  Project  1 1  has  18  comments.  Most  of  the  comments  indicate 
that  this  project  takes  the  wrong  approach  to  the  problem,  is  simplistic  and 
vague.  Only  in  one  of  the  18  comments  does  the  expert  see  this  project 
addressing  an  important  problem  that  can  be  solved.  Project  1 1  is  properly 
ronked  by  the  OSS  and  also  should  not  be  considered  further. 

Group  3  Project  7  has  12  comments.  Only  two  comments  are  clearly 
negative  while  six  comments  suggest  that  this  project  very  important  and 
achievable.  Many  of  the  comments  have  mixed  content  such  that  the  project 
is  recognized  as  important  but  the  experts  feel  that  it  is  being  addressed  by 
other  means  or  that  the  project  description  lacks  content.  With  such  on 
overall  agreement  on  the  importance  of  Project  7  and  general  agreement 
that  it  is  achievable.  Project  7  should  be  considered  for  further 
development. 

Example  Sensitivity  Analysis.  This  DSS  is  intend  for  managerial  use. 
The  manager  can  perform  a  sensitivity  analysis  simply  by  shifting  the 
weight  on  any  of  the  criteria  to  see  what  effect  it  has  on  the  decision 
results.  For  example,  if  the  monager  decides  that  criterion  5  in  the  Group  1 


top  five  DEM  is  twice  as  important  os  the  other  four  criteria,  the  weight  is 


changed  to  two  and  the  decision  result  is  then  reevaluated.  This  result  is 
shown  in  Figure  14. 


Figure  14.  Example  Sensitivity  Analysis:  C5  Shifted  X2 


The  numeric  score  of  each  project  is  changed  as  a  result  of  shifting  C5 
to  twice  the  weight  of  the  other  criteria,  but  the  rank  of  each  project  has 
remained  the  same.  This  indicates  that  the  project  ranks  are  stable  with 
respect  to  small  changes  in  C5.  Figure  15  shows  the  results  of  a  large 
change  in  C5:  The  weight  is  changed  to  five  times  that  of  the  other  four. 
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Figure  15.  Example  Sensitivity  Analysis:  C5  Shifted  X5 


In  Figure  15,  the  numeric  scores  of  the  projects  reflect  a  change  in 
project  rank;  project  5  has  moved  from  third  rank  to  first  while  the  other 
four  hove  only  shifted  down  in  rank.  This  indicates  that  only  project  5  is 
sensitive  to  large  changes  in  the  weight  of  C5  and  the  other  four  remain 
relatively  stable  with  respect  to  the  others.  The  manager  con  shift  the 
weights  of  the  other  criteria  in  the  some  way  to  assess  the  sensitivity  of 
this  project  group  to  the  other  criteria,  in  addition,  the  numeric 
comparison.  Illustrated  In  Figure  6,  could  be  used  to  see  the  effect  that 
changing  the  criterion  weight  has  on  each  project  score.  However,  each  DEM 
must  be  analyzed  separately  and  If  a  shift  In  criteria  weight  causes  a 
change  in  project  rank  in  the  other  OEMs,  the  new  top  scoring  project  must 
be  reenterd  Into  the  top  five  project  OEM  for  further  analysis.  Thus,  the 
limit  of  five  alternatives  per  DEM  could  cause  the  sensitivity  analysis  of  a 
large  number  of  alternatives  to  be  an  iterative  process  consisting  of 
analysis  and  reentering  the  project  data  if  a  change  in  criterion  weight 


changes  the  result. 


Conclusion 

The  general  DSS  utility  been  discussed,  which  led  to  the  specific  DSS 
utility  and  exercising  the  DSS  with  actual  project  data.  Exercising  this  DSS 
with  the  A!  project  data  has  revealed  some  deficiencies  and  strong  points  in 
the  system.  These  points,  along  with  the  managerial  assessment,  are 
discussed  in  the  next  chapter. 


V.  Discussion  and  Conclusions 

Introduction 

Inherent  in  the  general  and  specific  utility  analyses  performed  in 
Chapter  IV  are  the  general  use  case,  and  the  specific  use  case  which  is  a 
single  situation  sub-set  of  the  general  case  and  utility.  In  other  words, 
there  are  many  decision  situations  and  data  types.  Illustrated  by  the  general 
decision  structure,  data  structure,  and  operation  steps,  for  which  this  DSS 
may  be  used.  The  specific  utility  relates  to  the  specific  situation  that 
results  when  one  decision  situation  and  data  type,  illustrated  by  the  Al 
project  selection  and  data,  is  applied  to  this  DSS. 

This  DSS  has  several  limitations  that  were  identified  through  exercising 
It  with  the  project  data  for  this  thesis.  These  limitations  are  first 
discussed  as  a  special  case  of  operation  and  use.  Some  of  the  criteria 
identified  in  Chapter  II  are  well  met,  and  this  is  discussed  next  as  a  general 
case  along  with  the  results  of  the  management  questionnaire.  Then,  some 
conclusions  are  drawn  from  the  lessons  learned  about  the  special  and 
general  cases.  Some  Ideas  for  future  DSS  research,  and  possibly 
development,  ore  presented  lost. 

The  Special  Case 

The  special  case  of  exercising  the  hardware  sub-system  and  DSS 
soft  wore  sub-system  with  the  A I  project  data  presented  a  unique  problem: 
The  Al  projects  were  scored  In  such  a  way  as  to  elicit  the  most  possible 
information  from  the  experts  about  the  projects,  including  their  comments 
which  comprise  an  integral  part  of  the  information.  A  DSS  is  needed  to 


assist  the  project  manager  not  only  with  the  structured,  numerical, 
analysis,  but  with  the  semi -structured,  expert  commentary,  analysis  also. 
The  special  case  of  the  Al  project  data  and  the  expert  commentary  as  it  is 
used  in  this  DSS  is  discussed  first. 

The  Expert  Commentary  and  ASP  Utility.  As  discussed  in  Chapter  III,  the 
Al  project  data  was  collected  through  a  complicated  methodology  that 
resulted  in  numeric,  quantifiable,  scores  as  well  expert  comments,  or 
non-quontifiable  scores.  The  methodology  was  designed  to  provide  data 
superior  to  that  normally  associated  with  project  selections. 

The  data  is  incorporated  into  this  DSS  on  a  strictly  numeric  basis.  This 
required  a  great  deal  of  data  preparation,  which  is  discussed  later,  and  the 
development  of  a  technique  for  alerting  the  decision  maker  when  the  expert 
commentary  could  significantly  alter  the  DSS-suggested  numeric-only 
decision  result.  The  technique  devised  is  the  aggregrate  standard 
deviation  (ASD).  As  indicated  in  the  ASD  analysis  in  Chapter  IV,  it  provides 
on  indication  of  overall  numeric  score  dispersion  on  each  project  by  way  of 
the  stondard  deviations  of  the  individual  criteria  scores  for  each  project; 
comments  were  solicited  from  those  experts  that  scored  a  project 
significantly  different  from  the  group  mean  and  therefore  aided  in  the 
resultant  high  standard  deviation.  Therefore,  the  ASD  provides  a  numeric 
indicator  of,  or  a  signal  alerting  the  decision  maker  to  review,  the 
non-quontifiable  expert  commentary.  The  ASD  provides  the  benefit  of  the 
expert  commentary  without  the  task  of  reviewing  every  comment  for  each 
project.  Only  those  comments  that  are  significant  os  indicated  by  a 
relatively  high  ASD  need  be  reviewed.  In  addition,  a  connection  to  the 
information  provided  by  the  expert  commentary  would  not  be  provided  by 


this  OSS  without  the  ASD  due  to  the  almost  non-existent  text  capability  of 
Decisionhop.  This  limitation  is  discussed  more  later  but  means  that  the 
expert  commentary  relating  to  a  specific  project  must  be  looked-up  in  the 
project  report  (12)  by  the  decision  maker.  The  ASD-expert  commentary 
connection,  or  text-to-OSS  capability  of  this  system,  leods  to  some  specific 
limitations. 

Three  areas  of  limitation  are  apparent  as  a  result  of  the  special  case,  or 
exercising  this  system  with  the  project  data:  The  number  of  alternatives, 
the  data  preparation/entry,  and  the  text  capability. 

Number  of  Alternatives.  The  number  of  alternatives  allowed  by 
DecIsionMap,  five  in  each  DEM,  is  the  first  limitation  encountered.  This  will 
not  permit  a  complete  decision  analysis  that  involves  more  than  five 
alternatives  to  be  performed  in  one  model  and  necessitates  the  building  of 
multiple  OEMs  as  in  the  specific  utility  onolysis.  One  method  that  eases 
this  problem  is  to  build  a  blank  DEM,  called  a  template,  which  is  used  for 
each  set  of  alternatives  and  just  enter  the  data  and  alternatives  for  each 
part  of  the  analysis.  However,  this  still  does  not  provide  anolysis  of  the 
whole  problem  In  a  single  model.  In  addition,  approximately  one  hour  is 
required  to  build  the  template  and  enter  the  data,  by  hand,  for  an  analysis  of 
a  group  of  15  or  20  projects  as  was  performed  on  the  Al  project  data.  This 
does  not  include  any  data  preparation  such  as  calculating  stondord 
deviations  or  ASDs.  A  project  manager  would  not  have  the  time  nor  the 
inclination  to  perform  these  tasks.  Thus,  an  analyst  would  almost  always  be 
needed  for  DEM  preparation.  Only  in  the  case  of  an  occasional,  or  'sensitive' 
(e.g.  One  that  the  manager  wishes  to  keep  private),  analysis  with  five  or 
less  alternatives  and  little  or  no  data  preparation  would  a  manager  be  likely 


to  prepare  the  DEM  instead  Instead  of  having  it  prepared. 

As  a  last  point  on  the  number  of  alternatives,  every  change  requires  a 
trade-off:  If  the  maximum  possible  number  of  alternatives  was  increased 
to  50  or  20  or  even  10,  system  complexity  would  also  increase  while  ease 
of  use  and  speed  of  output  would  decrease.  For  example,  10  alternatives 
would  require  multiple  alternatives  list  screens  Instead  of  just  one,  a 
compressed  or  multiple  ranking  screens,  and  a  compressed  numeric/graphic 
output  similar  to  Figures  9, 1 1  and  13  but  with  all  10  alternatives 
presented  instead  of  only  five.  In  addition,  10  alternatives  would  require 
approximately  two  times  the  calculations  required  for  five  alternatives, 
slowing  the  output  response  speed  and  forcing  the  decision  maker  to  wait 
for  the  output. 

Dote  Preparation/Entry.  Data  preparation  may  range  from  easy,  requiring 
no  chonge  from  Its  raw  form,  to  difficult,  requiring  financial  or  statistical 
manipulation.  Much  of  the  statistical  data  preparation  for  the  project 
analysis  was  performed  and  Is  presented  in  the  project  report  (12:A-39  to 
A-42, 0-49  to  0-54,  and  C-47  to  C-50).  However,  if  the  mean  criterion 
scores  and  standard  deviations  were  not  available,  they  would  have  had  to 
hove  been  calculated  for  the  decision  analysis.  As  ft  was,  the  ASD  for  each 
project  hod  to  be  hand  calculated  and  entered,  as  well  as  multiplying  and 
rounding  the  mean  criterion  scores  as  described  in  Chapter  IV.  This  process 
odded  another  half  hour  to  the  preparation  for  each  DEM.  Again,  a  project 
monoger  is  not  likely  to  perform  this  type  of  data  preparation. 

The  data  manipulation  described  above  and  in  Chapter  IV  results  in  a 
loss  of  some  versatility  since  the  row  project  scores  are  not  used.  For 
example,  if  the  Likert  scale  scores  and  the  scale  weights  were  entered  in  a 


table  format  from  which  the  DSS  could  read,  the  decision  maker  could 
review  them  to  see  if  there  is  a  split  opinion  causing  a  high  standard 
deviation  instead  of  depending  on  the  ASD,  etc.  A  split  opinion,  or  bi-modal, 
indicator  could  be  programmed  into  the  DSS  to  alert  the  project  manager  if 
the  expert  commentary  should  be  reviewed  instead  of  requiring  the  project 
manager  to  visually  review  the  ASD  along  with  the  project  rank  to 
determine  if  the  comments  should  be  reviewed.  In  addition,  the  criteria 
score  standard  deviations  were  previously  calculated;  If  the  DSS  had  o 
statistical  calculation  capability  the  standard  deviations  and  the  ASD  could 
be  computed  directly  from  the  dato  and  used  in  the  DSS.  However,  both 
modifying  DecfsfonMap  and  linking  it  to  other  software  are  not  possible,  as 
mentioned  in  Chopter  IV. 

Two  additional  points  about  the  Al  dota  and  its  preparation  are 
noteworthy.  First,  If  the  Al  project  dato  had  not  been  available  it  would 
hove  been  necessary  to  find  other,  and  possibly  less  complete,  project 
selection  data  with  which  to  test  this  DSS.  This  means  thot  appropriate 
project  dota  would  hove  to  hove  been  located,  and,  if  not  already  available 
with  the  dato,  the  decision  criterlo  would  hove  to  be  established  for  a 
decision  model  and  a  ranking  methodology  for  the  projects  on  the  criteria 
would  hove  to  be  developed.  Second,  in  all  likelihood  only  a  numeric 
decision  onalysis  could  be  performed  since  expert  commentary  does  not 
usually  accompony  project  scoring  data. 

The  dato  entry  is  the  easiest  part  of  this  process;  a  bar  graph  scole  is 
moved  by  the  Macintosh  mouse  to  the  appropriate  number  (Operation  Step  6, 
Diopter  IV)  for  the  alternative  and  the  ASD  is  typed  into  the  comments 
space  provided  for  each  project.  However,  this  becomes  laborious  with  a 


large  number  of  alternatives  and/or  a  large  number  of  decision  criteria  and 
It  took  approximately  an  hour  and  a  holf  total  for  data  preparation  and  entry 
for  each  project  group.  An  analyst  Is  needed  for  large  decision  projects  of 
this  type. 

Finally,  as  mentioned  above,  there  a  trade-offs  Involved  with  every 
Increase  In  system  power  or  flexibility.  If  the  raw  scores  were  used  In  this 
decision  situation  there  would  be  251 1  numbers  to  enter  (Chapter  III), 
rather  then  the  279  mean  scores.  This  Is  not  only  more  work,  and  nine  times 
the  possibility  for  more  errors  In  data  entry  but,  would  require  a  link 
between  the  data  and  the  DSS  and  would  require  the  DSS  to  perform 
additional  calculations  to  arrive  at  the  decision  results,  both  of  which 
would  slow  the  output  ond  reduce  userfriendllness.  In  addition,  a  direct  link 
to  the  dota  Indicating  a  bl -modal  distribution  would  result  in  an  Increased 
complexity  of  DEM  building,  decreased  manager  userfriendllness  and  only 
further  ensuring  that  the  manager  would  not  be  Involved  In  the  DEM  building 
process. 

Text  Capability.  Most  project  assessment  date  do  not  Include  expert 
comments,  or  a  qualitative  assessment,  associated  with  attributes  of  the 
numeric,  or  quantitative,  assessment  as  does  the  Al  project  data  descrjed 
In  Chapter  III  and  discussed  above.  The  ‘richness"  of  the  Al  project  data 
would  be  lost  without  the  expert  commentary.  Therefore,  the  method 
developed  with  the  ASD  provides  o  ’flog"  for  the  manager  to  review  the 
expert  commentary,  but  they  must  be  looked  up  In  the  Al  project  report. 

This  Is  not  a  hard  process,  and  the  necessity  may  not  arise  often  In  project 
selection,  but  In  this  cose  text  handling  capability  would  have  been 
beneficial.  It  would  be  easier  to  Incorporate  all  of  the  comment  data  in  the 


DSS  for  review  by  the  manager  when  desired,  in  addition,  the  name  length  of 
20  characters  for  the  alternatives  and  decision  criteria  and  50  characters 
for  the  alternative  comments  causes  a  restriction  on  the  descriptive  nature 
of  names  and  comments  that  can  be  used.  For  example,  'Consistency  With 
Overall  Goals/Objectives’  (12:A-5)  would  be  preferable  to  Cl  as  a  criterion 
name  and  would  facilitate  sensitivity  analysis  by  clearly  indicating  which 
criterion  weight  Is  being  changed  rather  than  having  to  know  each  criterion 
or  look  It  up.  In  the  same  way,  Proj  1  for  'Al  Configuration  Design  of 
Modular  Fixing  Based  on  System  (12:A-7)  does  not  provide  the  decision 
maker  any  information  about  the  specific  project.  On  the  other  hand,  short 
non-descrlptlve  names  prevent  bias  from  entering,  either  Intentionally  or 
unintentionally.  Into  the  criterion  weights  or  project  ranks.  As  discussed 
above,  every  Increase  in  power  results  in  a  trade-off  and  in  this  case  the 
trade-off  includes  increased  name  length  in  trade  for  an  increased 
possibility  of  bias  and  a  decrease  in  ease  of  use,  possibly  by  hovlng  multiple 
nome  entry  screens  to  accommodate  the  Increased  name  length. 

The  General  Case 

The  areas  discussed  above  Indicate  the  limitations  revealed  through  the 
special  case  of  exercising  the  system  for  this  project.  The  general 
situation  assessment  is  a  result  of  demonstrating  this  DSS  to  managers  for 
their  perceptions  of  DSS  utility.  As  argued  in  Chapter  Ilf,  an  assessment  of 
managerial  satisfaction  with  this  DSS  may  be  used  as  a  measure  of  DSS 
utility  (8.785).  This  DSS  was  demonstrated  to  1 1  monogers,  ten  of  which 
were  either  military  or  civil  service  employees  Involved  in  Air  Force 
project  management  and  one  civilian  involved  in  DSS  development.  Most  of 
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the  demonstrati  on/assessment  sessions  were  one-on-one,  with  only  the 
manager  and  researcher  present,  but  in  one  session  three  managers  were 
present  and  in  another  two  managers  were  present.  As  described  in 
Chapter  II,  the  demonstrations  were  considered  "hands-on"  since  each 
manager  had  the  opportunity  to  manipulate  the  DSS  after  the  DSS  operation 
was  explained  to  them.  The  demonstrations  were  accomplished  by  carrying 
the  system,  comprising  of  the  microcomputer  and  software,  to  each 
manager's  office.  The  hardware  set-up  only  required  connecting  the 
keyboard  and  mouse  to  the  computer  ond  plugging  the  computer  into  a  wall 
outlet.  An  verbal  overview  of  this  research  effort  was  provided  to  eoch 
manager,  after  which  the  specific  case  of  the  Al  projects  was  explained  and 
the  demonstration  of  this  DSS  began  with  starting  the  software  as 
explained  in  Step  1  (Operation  Steps,  Chapter  IV)  and  progressed  through 
Step  7,  decision  results  and  sensitivity  analysts.  Each  demonstration  lasted 
approximately  20  minutes,  followed  by  an  opportunity  for  the  manager  to 
manipulate  the  system  for  approximately  40  minutes;  in  the  cases  where 
more  than  one  manager  was  present  the  hands-on  session  was  extended  to 
os  much  as  one  ond  one-half  hours.  At  the  end  of  the  session  each  manager 
completed  the  management  questionnaire,  developed  in  Chapter  III,  which 
asked  the  manager's  perceptions  about  this  DSS  with  respect  to  the  areas  of 
the  decision  environment,  output  ond  speed,  ease  of  preparation, 
userfriendliness,  and  overall  satisfaction  with  the  system.  The  mean 
scores  on  the  questionnaire  questions  are  in  Table  IV  along  with  a 
representation  of  the  questionnaire;  eoch  question  was  scored  on  a  five 
point  Likert  scale  where  o  score  of  three  on  a  question  indicates  no  opinion. 


Table  IV. 

Management  Questionnaire  Mean  Scores 
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Use  a  number  from  the  scale  below  to  Indicate  your 
agreement/disagreement  with  questions  1-15. 

NO  VES 


Mean 

Question  Score 


1. 

Is  sufficient  data  incorporated  for  this  to  be  a  useful  decision  aid? 

4.27 

2. 

Could  sufficient  data  be  incorporated  for  future  decisions? 

4.64 

3. 

Does  this  decision  aid  react  reasonably  when  manipulated? 

4.80 

4. 

Does  this  decision  aid  simulate  real  world  relationships? 

4.18 

5. 

Does  this  decision  aid  provide  an  understandable  output/result? 

4.55 

6. 

Is  the  output/result  relevant  to  the  decision? 

4.55 

7. 

Is  the  decision  aid  operation  sufficiently  fast? 

4.64 

8. 

Do  you  think  that  future  decision  aid  preparation  would  be  easy? 

3.82 

9. 

Do  you  think  that  future  data  preparation  would  be  easy? 

3.50 

10. 

Is  the  system  (computer  and  software)  easy  to  use? 

4.73 

11. 

Is  the  computer  easy  to  use? 

4.82 

12. 

Is  the  software  easy  to  use? 

4.73 

13. 

Would  this  decision  aid  improve  your  decision  making? 

4.18 

14. 

Would  you  use  this  system? 

4.27 

15. 

Would  you  recommend  use  of  this  system? 

4.27 

16. 

Comments  or  recommendations? 

;  s’ 


one  indicates  a  negative  response  and  five  indicates  a  positive  response.  In 
addition  to  the  responses  in  the  above  areas,  the  managers  were  asked  for 
comments  and  recommendations  and  they  identified  training  and  the 
Macintosh  hardware  subsystem,  which  is  not  Air  Force  standard  equipment, 
os  problem  areas.  Training  and  hardware  ore  discussed  later  os  they  relate 
to  the  broader  issues  of  the  general  case  os  opposed  to  the  more  narrow 
scope  of  the  specific  cose. 

In  summary,  the  questions  ore  designed  to  elicit  a  manager's  perceptions 
with  respect  to  this  DSS  about  how  well  it  fits  the  decision  environment 
(Questions  1-4),  the  output  and  speed  (Questions  5-7),  ease  of  preparation 
(Questions  8,9),  and  userfriendliness  (Questions  10-12).  Although  the 
number  of  managers  surveyed  is  not  large  enough  to  be  valid  for  a 
statistical  analysis,  a  preliminary  indication  of  whether  or  not  this  DSS 
aids  the  decision  process  can  be  derived.  A  discussion  of  these  results 
follows.  The  raw  scores  and  comments  ore  In  Appendix  C. 

The  Decision  Environment.  In  on  effort  to  represent  the  decision 
environment  this  DSS  was  assembled  to  simulate  one  type  of  decision 
trade-off  that  project  monagers  make  for  project  selection,  a  decision  tree 
as  was  found  by  Prince  (15).  This  includes  the  amount  of  data  incorporated 
Into  the  DEM  and  how  the  DEM  reacts  when  manipulated,  os  well  as  the  real 
world  decision  relationships  that  the  DSS  that  is  intended  to  support. 
Questions  1  and  2  on  the  management  questionnaire  ask  about  the  amount  of 
doto  used  in  the  project  DEMs  and  the  amount  of  data  that  may  be 
incorporated  in  future  DEMs.  The  amount  of  project  DEM  data  ore  considered 
sufficient  with  a  mean  question  score  of  4.27,  but  the  limit  of  five 
alternatives  per  DEM  has  an  impact  on  this  question;  over  half  of  the 


managers  felt  that  this  limit  is  a  problem.  Others  felt  that  it  could  be 
adequately  worked  around  or  they  do  not  routinely  have  more  than  five 
alternatives  to  choose  among.  Very  large  OEMs,  of  125  factors  on  Level  3 
and  625  factors  in  Level  4,  can  be  built  and  are  limited  in  size  only  by  the 
memory  in  the  computer.  The  managers  recognized  this;  the  mean  score  for 
question  2  is  4.64  which  Indicated  a  high  degree  of  agreement  with  the 
question  of  sufficient  data  for  future  OEMs.  However,  a  decision  requiring 
even  100  separate  factors  would  be  incomprehensible  to  most  people;  it 
would  need  to  be  divided  into  smaller  parts  that  could  be  easily  managed  and 
understood  by  the  decision  maker.  There  were  no  comments,  verbal  or 
written,  about  the  amount  of  data  but  one  manager  did  comment  that  the 
link  to  a  data  base,  and  the  corresponding  functions.  Is  a  necessity  that  is 
missing. 

Question  3  asked  if  the  DSS  reacts  reasonably  when  manipulated;  this  is 
on  effort  to  determine  If  the  managers  understood  the  decision  model  and 
what  it  does  when  the  weights  of  the  criteria  are  shifted.  The  mean 
question  score  Is  4.00  which  indicates  a  high  degree  of  agreement  with  the 
question.  The  only  comment  on  this  question  suggests  that  automated 
sensitivity  analysis,  by  computerized  subroutines,  would  be  a  help. 

The  heart  of  the  DSS  Is  its  ability  to  simulate  the  real  world 
relationships  that  the  manager  must  deal  with  to  moke  a  project  selection. 
Question  4  asks  this  and  the  managers  responded  with  a  mean  question  score 
of  4.10.  Written  and  verbal  comments  reveal  that  the  monogers  feel  that 
real  world  decisions  are  less  structured  and  require  more  Interaction  with 
people,  with  the  emphasis  on  people  making  decisions,  not  machines.  This 
same  point  is  mode  by  both  Davis  (4:368-369)  and  North  (13:200).  However, 


the  mean  question  score  indicates  that  the  managers  answers  tend  more 
toword  the  positive  than  a  neutral  or  negative  response. 

DSS  Output  and  Speed.  Questions  5  and  6  deal  specifically  with  the 
understandability  and  relevancy  of  the  DSS  output.  For  example,  the 
managers  were  shown  the  numeric/graphic  outputs  os  in  Figure  9  and  text 
outputs  as  in  Figure  0  which  show  the  rank-ordered  alternatives,  as  well  as 
the  decision  analysis  outputs  os  in  Figures  6  and  7.  The  mean  question  score 
on  both  questions  is  4.55  which  indicates  that  most  managers  understood 
the  outputs  and  that  they  are  sufficient  for  the  project  selection  type  of 
decision  situation.  The  text  capability  was  not  identified  as  a  problem  in 
the  general  case  as  it  was  in  the  special  case;  sufficiently  descriptive 
alternative  and  decision  factor  names,  for  general  use,  can  be  entered  into 
DecIsionMap’s  20  character  length  name  limit.  Verbal  comments  Indicated 
that  the  graphical  outputs  are  much  more  preferred  by  the  managers  than 
the  long  lists  of  numbers  that  they  are  accustomed  to  receiving  from  other 
computer  applications,  in  addition,  the  DSS  response  when  the  decision 
results  are  requested,  or  when  a  criterion  weight  Is  changed  for  a 
sensitivity  analysis,  is  almost  Instantaneous.  "Speed  of  response’  Is  an 
Important  DSS  attribute  Identified  by  Wagner  (22:78).  Question  7  asked  if 
the  DSS  is  sufficiently  fast  and  the  question  mean  score  is  4.64,  which 
Indicates  that  the  managers  felt  that  the  outputs  are  presented  sufficiently 
fast. 

DSS  Preparation.  The  DEM  and  data  preparation  hove  been  discussed 
previously.  Most  of  this  Information  (With  the  exception  of  the  ASD  which 
had  not  been  included  in  the  DEM  when  the  demonstrations  were 
accomplished)  was  discussed  with  the  managers.  The  menegers  all  felt  that 
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the  problem  definition,  building  o  proper  decision  model,  end  then  preporing 
the  date  are  the  most  difficult  parts  of  a  decision  analysis.  This  is 
reflected  by  the  mean  question  scores  of  3.02  ond  3.50  on  Questions  6  and  9 
which  ask  if  the  managers  think  that  future  decision  aid  preparation  and 
data  preparation  would  be  easy.  The  subject  of  DEM  and  data  preparation 
evoked  the  second-largest  number  of  verbal  and  written  comments.  For 
example,  two  managers  felt  that  the  process  ef  building  the  DEM  and 
entering  the  data  on  the  computer  is  almost  trivial  but  that  defining  the 
problem  and  the  decision  criteria  before  building  the  DEM  is  often  an  almost 
Impossible  task.  Another  manager  preferred  computer  simulation 
techniques  for  "Selection  and  structuring  of  the  decision  model".  One 
manager  noted  that  this  DSS  only  manipulated  the  data,  that  data  gathering 
and  preparation  is  the  hard  part.  However,  the  managers  felt  that  the 
decision-tree  format  provided  by  DecisionMap  does  aid  the  process  of 
converting  their  mental  decision  trade-off  to  a  more  structured  and 
understandable  decision  format,  even  if  defining  the  problem  and  gathering 
the  data  are  the  hardest  parts  of  the  decision  onalysis. 

Usarfriendliness.  "Perhaps  the  single  most  vital  criterion"  (17:23)  for  a 
DSS  intended  for  managerial,  and  possibly  infrequent,  use  is 
usarfriendliness.  This  subject  provided  the  greatest  number  of  comments, 
both  verbal  and  written.  Thus,  Questions  10  -12  ask  the  manager's 
perceptions  about  the  system,  computer,  ond  software  userfriendliness.  The 
sub-system  components  ore  separately  identified  in  these  questions  in  a 
effort  to  see  if  the  managers  perceive  a  distinct  difference  in  the  computer 
subsystem  operation  (Macintosh)  os  opposed  to  the  application  software 
subsystem  operation  (DecisionMap)  and  the  operation  in  combination,  or  the 


system  os  a  whole.  A  single  question  about  the  system  userfriendllness 
could  hide  the  perception  that  the  hardware  Is  easy  to  operate  while  the 
software  Is  confusing,  or  vice  versa.  However,  this  Is  not  the  case  since 
Questions  10  and  12  both  hove  a  mean  question  score  of  4.73  (On 
userfriendllness  of  the  system  and  software,  respectively)  and  Question  1 1 
has  o  slightly  higher  mean  score  of  4.82  (On  userfriendllness  of  the 
computer).  The  managers  agree  that  this  Is  the  most  userfrlendly  system 
that  they  have  encountered;  oil  but  one  felt  that  the  Icon/mouse  operating 
system  Is  superior  to  menu-driven  command-oriented  computers.  Is  easier 
to  use,  and  provides  a  pleasing  graphics  Interface.  In  addition,  all  but  two 
managers  verbally  commented  that  they  felt  that  the  system  provided  an 
environment  that  olds  the  decision  process  through  the  visually  oriented 
process  of  building  decision  models  and  sensitivity  analysis.  The  manager's 
understanding  of  the  DEM  building  process  and  sensitivity  analysis  is 
discussed  in  the  section  on  training.  Finally,  the  managers  (Again  all  but 
one)  expressed  the  desire  for  this  type  of  userfrlendly  interface  to  be 
transported  to  other  microcomputers  and  applications  even  If  this  specific 
application  does  not  completely  fulfill  their  requirements. 

Overall  Satisfaction.  Rather  than  ask  each  manager's  opinion  about 
overall  satisfaction  with  this  system,  which  could  be  interpreted  in  many 
different  ways  due  to  the  complexity  of  the  system  and  the  one-on-one 
demonstration  scenario,  the  managers  were  asked  if  they  thought  this 
system  would  improve  their  decision  making  and  if  they  would  use,  ond 
recommend  use  of,  the  system.  Question  13  asks  if  the  managers  think  thot 
this  DSS  would  improve  their  decision  making;  the  question  mean  score  is 
4.18.  This  indicates  that  some  improvement  in  decision  making  may  be 


perceived  but  there  were  no  specific  comments,  other  thon  those  mentioned 
previously  on  the  number  of  alternatives  and  DEM  preparation,  about  this 
subject.  Questions  14  and  15  ask  if  the  manager  would  use  and  recommend 
use  of  the  system,  respectively.  Question  15,  asking  if  the  manager  would 
recommend  use  of  the  system,  was  included  to  add  additional  information  on 
how  satisfied  a  manager  is  with  the  system;  he  or  she  might  say  that  they 
would  use  the  system,  but  are  they  satisfied  enough  with  the  system  to 
recommend  its  use  to  someone  else?  The  mean  question  score  on  both 
questions  is  4.27  which  Indicates  that  the  managers  feel  the  same  about 
using  the  system  and  recommending  its  use.  Four  managers  thought  that 
they  could  put  the  system  to  immediate  use  on  current  decision  problems 
that  they  were  working  on  if  they  had  it  available.  All  but  two  of  the 
managers  responded  vertally  that  this  DSS  would  be  an  aid  for  improved 
decision  making  in  some  situations,  but  most  wanted  the  capability  for 
more  alternatives,  as  Indicated  previously.  Finally,  some  managers  were 
'overwhelmed*  by  the  system  os  a  whole  since  it  is  so  drastically  different 
from  any  that  they  had  seen  before.  They  would  hove  liked  more  time  to 
learn  and  use  the  system. 

Training.  The  objective  of  the  managerial  demonstrations  was  for  the 
manager  to  learn  the  basic  operation  of  this  DSS  and  put  it  to  immediate 
use.  Two  points  were  stressed  for  the  managerial  demonstrations:  First, 
the  monogers  do  not  have  much  time  to  give  for  testing  a  system  and  second, 
the  system  is  torgeted  for  managerial  use  with  minimum  training  end 
therefore  exhibits  extreme,  almost  intuitive,  ease  of  use.  Every  manager 
quickly  caught  on  to  the  system  operation,  DEM  building  and  sensitivity 
analysis,  and  decision  result  manipulation.  However,  they  had  trouble 


reloting  to  a  decision  situation  that  they  were  not  familiar  with  before  hand 
(Such  as  the  Al  projects)  with  the  result  that  some  managers  built  a  DEM  for 
a  decision  that  they  were  currently  working  on  and  therefore  familiar  with. 
The  other  managers  preferred  to  exercise  the  system  with  other  pre-built 
examples  on  such  things  os  buying  a  car  or  house.  This  illustrates  the  need 
to  be  familiar  with  a  specific  problem  before  a  rational  decision  analysis 
can  be  performed.  In  addition,  every  manager  felt  thot  they  did  not  hove 
enough  time  with  the  system  to  fully  judge  its  capabilities  and  weaknesses. 
Although  the  brief  training  demonstration  was  sufficient  to  get  the 
managers  started,  an  extended  amount  of  time  with  the  system  is  required 
for  a  better  analysis  of  it  and  therefore,  on  improved  feedback  for  future 
DSS  reseorch. 

Conclusion  and  Recommendations 

The  reseorch  objective  wos  to  locate,  acquire,  demonstrate  realistically, 
ond  assess  a  commercial  microcomputer  decision  support  system  thot 
supports  ond  aids  the  R&D  project  manager's  semi -structured  project 
selection  decision  process.  The  research  objective  resulted  in  two 
investigative  questions: 

1.  What  ore  the  semi -structured  decision  processes  that  Air  Force  R&D 
project  managers  use  now  when  assigning  program  priorities?  This 
question  was  answered  by  two  previous  researchers,  Lee  (1 1)  ond 
Prince  (IS),  and  was  found  to  be  checklists  and  profile  charts,scoring 
models,  decision  trees,  and  goal  programming.  A  project  manager  faced 
with  the  particular  decision  situation  of  deciding  which  alternative  project 
(or  technology)  to  pursue,  using  one  or  more  of  these  techniques,  is  termed 


to  be  operating  in  the  decision  environment. 

2.  What  commercial  microcomputer  decision  aid  best  fits  the  specific 
DSS  criteria  and  will  aid  this  R&D  semi-structured  decision  environment? 
This  question  required  multiple  steps  to  arrive  at  an  answer. 

First,  the  composition  of  a  computer  based  decision  support  system  was 
determined  to  be  the  hardware  sub-system,  the  operation  software 
sub-system,  and  the  application  software  subsystem. 

Second,  the  choracteri sties  and  functions  of  a  DSS  were  found  in  the 
literature.  These  characteristics  and  functions,  along  with  some  research 
constraints,  were  used  to  develop  12  specific  criteria  for  this  managerial 
DSS. 

Third,  the  hardware  sub-system  was  selected  on  the  basis  of  research 
constraint  number  three  (criteria  number  10  in  Table  II)  and  the  specific 
DSS  criterion  that  dictates  userfriendliness  for  a  managerial  DSS.  Although 
userfrlendliness,  like  beauty,  is  in  the  eye  of  the  beholder,  the  hardware 
sub-system  trade-off  revolved  around  a  typed-command  keyboard-entry 
computer  (IBM  or  compatibles)  thot  is  widely  available  in  the  Air  Force  and 
a  mouse  operated  visually-oriented  computer  (Apple  Macintosh)  thot  is  much 
less  available  in  the  Air  Force  but  requires  little  typing  to  operate  and 
simulates  the  manager's  desk-top  working  area.  The  Macintosh  was  chosen 
due  to  research  constraint  number  three  and  In  a  specific  effort  to  ossess 
manager's  perceptions  of  "userfriendliness". 

Fourth,  microcomputer  software  that  in  some  way  fit  the  decision 
environment  (Not  just  DSS  software  but  microcomputer  software  that  aids 
one  of  the  techniques  Identified  by  Prince)  was  evaluated  apainst  the  12 
criteria  developed  previously.  DecisionMe?  met  1 1  of  12  criteria  and  was 
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chosen  os  the  best  alternative,  with  price  being  the  deciding  factor. 

Fifth,  a  realistic  demonstration  was  accomplished  through  a  specific 
utility  analysis  Involving  actual  project  data  and  a  general  utility  analysis 
involving  project  managers  operating  this  DSS.  The  specific  utility  analysis 
included  structured  (numerical)  analysis  and  semi -structured  (expert 
commentary)  analysis.  The  general  utility  analysis  concluded  with  the 
managers  providing  their  perceptions  of  this  DSS  on  a  questionnaire. 

Thus,  a  microcomputer  decision  aid  that  fits  the  specific  DSS  criteria 
and  aids  the  R&D  semi -structured  project  selection  decision  process  has 
been  located,  acquired,  and  realistically  demonstrated.  But,  is  this  the 
"best"  decision  aid?  The  results  of  the  general  and  specific  utility  analyses 
indicate  that  more  functions  are  needed  while  retaining  the  Macintosh- type 
of  user  interface.  Some  additional  functions  are  discussed  more  in  the 
section  on  recommendations  for  future  research. 

Is  there  a  commercial  microcomputer  decision  support  system  thot  will 
aid  the  Air  Force  project  manager  in  project  selection?  Possibly,  but  this 
DSS  does  not  provide  a  complete  answer.  The  areas  discussed  previously 
indicate  some  limitations  and  some  strengths  of  this  DSS.  The  monagers 
feel  that  this  DSS  Is  a  step  In  the  right  direction  for  userfriendliness  and 
modeling  the  decision  environment,  but  it  would  be  more  useful  os  a 
decision  aid  if  it  could  handle  more  than  five  alternatives  at  a  time  and  had 
a  link  to  a  data  base.  In  addition,  o  word  processing  capability,  or  extended 
comment  pages  for  each  alternative,  would  help  improve  its  capabilities. 
However,  this  DSS  provides  an  excellent  system,  for  the  price,  for  decisions 
involving  interrelated  decision  factors  with  five  alternatives  or  less.  It  is 
extremely  userfriendly,  fast,  and  aids  the  visualization,  understanding,  and 
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analysis  of  many  decision  problems  within  its  capabilities.  The  second 
place  DSS  software  in  Table  II,  Chapter  III,  does  provide  some  additional 
capabilities  at  almost  three  times  the  price.  However,  this  Introduces 
another  problem:  Software  product  sheets  often  indicate  that  the  product 
can  accomplish  almost  onything,  which  is  obviously  not  the  case,  and 
commercial  software  developers  are  very  reluctant  to  answer  specific 
questions  over  the  telephone.  Every  commercial  softwore  developer 
contacted  with  respect  to  this  research  said  they  never  supply  evaluation 
softwore  for  academic  reseorch  projects.  Therefore,  there  is  virtually  no 
way  to  adequately  evaluate  the  capabilities  of  different  DSS  software  short 
of  purchasing  eoch  program  and  evaluating  each  one  individually.  Some 
recommendations  for  the  evaluation  of  DSSs  is  presented  next. 

Improved  DSS  Criteria:  A  System  Specification.  Before  additional  DSS 
research  is  performed,  an  improved  list  of  DSS  criteria  is  needed.  It  Is 
recognized  that  any  one  of  the  following  criteria  result  in  system 
trade-offs  of  increased  system  capabilities  with  Increased  system 
complexity  and  decreased  ease  of  use,  and  the  proposed  manager-user  should 
be  kept  os  a  reference  point  at  all  times.  The  DSS  criteria  should  evolve 
from  necessary  system  characteristics  to  a  system  specification  that  can 
be  used  for  an  assessment  of  DSS  capabilities,  either  for  commercial 
softwore  evaluation  or  for  future  DSS  softwore  development.  Then  an 
extended  managerial  assessment  (Discussed  more  in  Training  Sufficiency) 
should  be  conducted  to  determine  system  utility  and  how  well  the  system 
specification  is  met.  Some  examples  of  the  system  characteristics  and 
specifications  required  ore: 

1.  The  system  should  provide  a  user  Interface  that  the  manager  con 

74 


M  ''iiV'*  ; 


easily  leem  end  use  (10;  3).  This  moy  be  provided  by  either  e  desk-top 
pictorlel  interface  with  which  the  manager  is  already  intuitively  familiar 
or  an  Improved  command-keyboard  interface  that  compensates  for  Inferior 
typing  skills  and  eliminates  the  confusion  associated  with  microcomputer 
operation  (10:24). 

2.  One  or  more  of  the  techniques  identified  by  Prince  (15)  should  be 
incorporated  as  the  basic  DEM  structure.  Depending  on  the  method(s)  used  a 
graphic  representation  of  the  DEM  should  be  provided  to  facilitate 
understanding  of  the  relationships  of  the  decision  factors.  In  addition,  the 
DEM  should  provide  an  ‘elimination  rule*  capability  such  that  a  specific 
decision  criterion  con  be  used  os  a  go/no-go  decision  point  (Such  os  an  over 
budget  cost  estimate),  provide  easily  weighting  of,  and  easy  changing  of 
weights  on,  decision  criteria,  and  provide  eosy  ranking  of  alternatives  on 
the  decision  criteria  (22:77,78).  Like  userfriendliness,  "easy"  is  in  the  eye 
of  the  beholder;  The  determination  of  whet  Is  eosy  for  managerial  use  could 
be  the  subject  of  another  complete  research  project. 

3.  Alternative  ranking  on  the  decision  criteria  may  be  achieved  by 
ranking  them  directly  on  the  decision  criteria,  through  a  link  to  o  specific 
part  of  a  data  base  (Such  as  time,  cost,  potential  pay-off,  and/or  other 
decision  critical  resource  factors  on  o  data  base  form),  or  through  o 
spreadsheet  link  which  could  use  specified  spreadsheet  cells  as  alternative 
ranks  on  the  decision  factors. 

4.  There  should  be  an  ‘optimal  limit*  on  the  number  of  oltemotives 
allowed  in  the  DEM.  Again,  determination  of  an  optimal  limit  of  alternatives 
Is  the  subject  of  further  research  to  determine  the  number  of  alternatives 
with  which  project  monagers  routinely  work. 
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5.  The  DSS  should  provide  some  type  of  extended  text  hendling  capability 
that  can  be  linked  to  the  DEM  both  quantatively  and  qualatively  in  an  attempt 
to  assist  in  semi -structured  decision  analysis  (4:368-369;  13:200). 

6.  If  a  commercial  DSS  is  used,  modifiable  source  code  should  be 
purchased,  if  possible,  to  provide  DSS  tailoring  when  necessary. 

One  system  specification  not  addressed  is  the  specific  hardwore 
sub-system  of  a  DSS  intended  for  Air  Force  users.  As  mentioned  previously, 
the  managers  involved  in  the  managerial  assessment  were  dismayed  at  the 
fact  that  this  DSS  could  not  be  immediately  transferred  to  their 
microcomputers  and  put  to  use;  the  current  potential  for  this  specific  DSS 
to  be  widely  used  in  the  Air  Force  environment  is  small.  However,  o 
conversation  with  Captain  William  J.  Raissle  revealed  that  there  is  “A 
Macintosh  on  every  desk'  at  his  program  management  office.  After  a 
demonstration  of  this  DSS  like  that  provided  the  other  managers.  Captain 
Raissle  said  that  they  are  looking  for  applications  software  such  as 
DecisionMap  and  could  put  it  to  immediate  and  productive  use  for  source 
selections  (17).  Thus,  there  is  some  potential  for  use  of  this  DSS  and  other 
Macintosh-bosed  decision  aid  applications  in  the  Air  Force  environment. 

Recommendations  for  Some  Future  DSS  Research.  In  addition  to  the 
research  areas  indicated  previously,  some  future  DSS  research  projects 
could  fall  Into  four  different  categories:  Commercial  DSS  softwore, 
integrated  software,  self  integrated  software,  and  developing  DSS  software. 
Whichever  route  it  token,  on  easily  visualized  (As  opposed  to  the  numeric 
formulas  used  in  spreadsheet  programs)  decision  modeling  Interface,  such 
os  that  provided  by  DecisionMap,  should  be  the  central  core  of  the  DSS  and  It 
should  be  supplemented  with  the  capabilities  identified  in  the  additional 


requirements  end  illustrated  in  Figure  16.  The  ovals,  in  Figure  16,  represent 
DSS  capabilities,  or  software  module  sub-systems,  while  the  bottom  figure 
represents  the  user. 


Each  double-headed  arrow  illustrates  the  transfer  of  data  and  information 
to  and  from  the  corresponding  sub-system,  with  the  only  access  by  the  user 
to  the  OSS  sub-systems  being  through  the  userfriendly  user  interface.  In 
this  way  the  analyst-user  has  full  access  to  the  DSS  capabilities  but  the 
manager-user  only  needs  to  operate  the  decision  modeling  subsystem, 
through  the  user  Interface,  once  the  DEM  Is  built  and  data  are  entered. 

Commercial  DSS  Software.  Many  of  the  DSSs  identified  in  Table  I  have 
additional  capabilities  at  a  higher  price.  However,  as  pointed  out  above  it  is 


difficult  ot  best  to  obtoin  complete  information  about  a  product  without 
first  buying  it,  ond  a  sponsor  is  needed  for  software  purchases.  But,  many 
of  these  DSSs  may  be  worthy  of  further  analysis,  and  may  even  provide  a 
user  interface  similar  to  that  provided  by  the  Mocintosh/DecislonMap 
combination  on  the  command-driven  computers. 

Integrated  Software.  As  mentioned  previously  in  Chapter  Hi,  many  of  the 
new  integrated  software  packages  provide  the  features  illustrated  in 
Figure  16.  Software  such  as  Jazz  and  Lotus  1-2-3  do  not  provide  the 
graphic  DEM  building  and  sensitivity  analysis  that  is  suggested,  but  an 
add-on  user  interface  could  be  developed  to  provide  these  features. 
Therefore,  one  of  these  products  could  provide  a  multiple-function 
integrated  software  core  for  a  future  DSS. 

Self  Integrated  Software.  Many  software  products  individually  provide 
the  capabilities  required  of  a  OSS;  separate  products  exist  for  data  base 
management,  word  processing,  and  decision  modeling  that  operate  on  the 
same  microcomputer  through  a  userfriendly  Interface.  These  products  can 
be  linked  together  to  produce  an  operational  DSS.  Since  the  products  already 
exist,  an  integration-code  module  would  be  written  to  combine  the  product 
functions.  Hence  the  term  'self  integrated  software*. 

A  problem  with  the  "software  baseline*  Is  Involved  with  both  the 
Integrated  and  self  Integrated  software  projects.  Software  developers  are 
continuously  improving  and  updating  their  products  which  could  result  In  on 
incompatibility  problem  between  the  privately  developed  programs.  Thus, 
once  the  DSS  softwore  is  fully  integrated  the  numbered  software  versions 
with  which  the  DSS  operates  should  be  established  as  a  baseline. 
Establishing  this  softwore  baseline  rests  with  the  DSS  developer;  there  is 


no  easy  solution  to  the  problem  of  who  will  update  the  DSS  software  if  a 
future,  privately  developed,  software  update  causes  an  incompatibility 
problem  with  the  DSS  software.  However,  many  software  developers  are 
attempting  to  provide  a  “standard*  data  tronsfer  format  which  will 
minimize  the  compatibility  problem. 

Develop  DSS  Software.  As  a  last  resort  and  in  spite  of  the  case  made  for 
commercial  software  in  Chapter  I,  new  DSS  software  could  be  developed. 

The  DSS  software  should  be  capable  of,  and  provide  an  interaction  or  trading 
of  dato  between,  data  base  and  mathematical  manipulation,  word 
processing,  graphical  decision  modeling  and  sensitivity  analysis,  through  a 
userfriendly  interface.  The  heart  of  this  system,  as  in  those  above,  is  the 
decision  modeling  capabilities  and  user  interface. 

Training  Sufficiency.  The  training  for  future  DSS  projects  should 
encompass  two  objectives:  First,  manager-user  training  for  the  intended 
users  and  second,  analyst-user  training  for  the  person  that  builds  the 
decision  models  and  prepares  the  date,  if  necessary,  if  the  system  provides 
userfriendly,  graphic,  decision  modeling  and  sensitivity  analysis  the 
managerial  training  could  be  limited  to  o  demonstration  to  explain  the 
system  and  DEM  operation,  supplemented  with  a  short  text  manual  of 
commands  and  operations  on  how  to  perform  decision  onalysis  with  the  DSS 
and  how  to  access  the  data  base  and  word  processing  functions.  The 
manager  should  then  use  the  system  to  build  DEMs  and  perform  decision  and 
sensitivity  analysis.  An  extended  period  of  managerial  use,  perhaps  a  month 
if  system  ovailability  and  time  permit,  should  be  provided  to  accomplish  a 
more  complete  assessment  and  allow  for  the  initial  'amazement*  of  the 
system  to  wear  off  ond  the  actual  utility  of  the  system  to  be  realized.  The 


Appendix  A:  Expert  Commentary 


The  expert  commentary  for  the  only  the  top  five  projects,  as  ranked  by  this 
DSS,  from  each  group  is  presented  here.  A  complete  list  of  the  commentary 
can  be  found  in  the  At  workshop  report  (12). 

Group  1  Expert  Commentary.  The  top  five  Group  1  projects,  in  order 
from  top  ranked  to  bottom,  are  projects  14, 6, 5, 4,  and  10.  The  expert 
commentary  is  listed  below.  The  Groups  2  and  3  commentary  follow  Group  1 
and  are  also  in  ronk  order. 

Project  14. 

Still  hove  a  problem  with  including  “adaptive  control"  as  an  Al  issue  in 
recovering  from  changes  in  structure. 

Why? 

1-  Adaptive  control  Al 

2-  Changes  in  structure  require  more 

I  view  this  project  as  feasible  with  respect  to  integration  of  automated 
statistical  quality  control  within  a  flexible  manufacturing  system.  Process 
diagnosis  could  result  In  automatic  machine  compensation  to  keep  the 
process  in  acceptable  limits. 

The  creation  of  on  Inference  engine  should  be  straight  forward  and  the 
Inspection  knowledge  Is  well  defined  end  specific;  these  combine  to  moke 
this  a  very  feasible  project. 
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I  view  this  project  as  providing  very  Important  contributions  to  At  in 
manufacturing  at  the  Inspection  end.  The  project  will  be  limited  to  milling 
and  lathe  operations.  With  this  limitation,  I  view  the  technique  feasibility 
of  the  project  to  be  quite  high. 


Project  6.  - 

Project  is  ambitious  ond  technically  difficult. 

The  process  is  feasible  if  approached  as  a  shell  of  intelligence  that 
integrates  discrete  processes  through  a  communication  facility.  The 
real-time  definition  would  need,  h  this  case,  to  be  defined  in  terms  of 
minutes  or  seconds. 

I  did  not  change  my  2  on  feasibility  because  I  believe  that  it  will  be  very 
difficult  to "...  formulate  (swiftly)  an  alternative  plan"...  based  on  sensory 
input. 

Project  5. 

It's  not  clear  to  me  that  the  behavior  of  composites  is  well  enough 
understood  -  even  at  a  heuristic  level  -  to  moke  this  feasible  as  a  2.5  yeor 
project. 

Is  the  knowledge  all  there? 

There  is  not  adequate  explanation  about  the  performance  strategy  of  this 
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research  project. 


Impact  is  substantial  -  shows  up  wherever  tools  and  Al  both  are  useful  for 
different  parts  of  some  problem. 


No  comments. 


Project  4. 


Project  10. 

No  Comments. 

Group  2  Expert  Commentary.  The  following  are  the  top  five  Group  2 
projects,  and  the  associated  expert  commentary,  ordered  from  top  to 
bottom. 


Project  5. 

Cl:  The  author  states  in  milestones  "Determine  how  to  analyse  CAD  data  to 
generate  knowledge"  -  this  is  a  difficult  problem  in  its  own  right  and 
shouldn't  be  mixed  into  a  generative  planner. 

If  the  project  were  intended  os  exploratory  development  or  advanced 
development,  I  would  ogree  with  the  assessment,  but  I  don't  think  a  "basic 
research"  project  would  require  that  much  time. 

I  strongly  feel  that  the  project  is  feasible  since  processes  ore  known  today 
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except  they  reside  within  the  experience  of  the  process  planner.  If  design 
information  can  be  passed  from  engineering  to  manufacturing  in  terms  of 
features,  tolerances,  and  specifications,  it  is  possible  to  combine  this 
information  with  a  rule  based  system  to  create  process  plans. 

Generative  process  plonning  should  be  addressed  in  exploratory  range.  Other 
efforts  are  already  generating  results  and  potential  near  term  benefits. 

This  statement  is  so  general  that  it  would  not  contribute  to  knowledge  of  on 
expert  in  the  area  and  would  net  advance  payoff  a  significant  increment. 

The  project  is  too  far-out  and  too  general. 

I  have  revised  C5  and  C6  but  only  on  the  basis  a  3-D  solid  geometric  CAD 
system  that  is  a  real-time  system,  is  available  for  design  and  detailing  and 
dimensioning. 

C6:  no  change  -  not  enough  detail  to  evaluate  -  50  my  for  whom? 

To  accomplish  objective  will  result  in  significant  step  in  feature 
recognition  and  thus  contribution  to  Al. 

This  project  is  so  global,  so  poorly  defined,  that  it  has  no  value  in  its 
present  form. 

Approach  not  clear.  If  it's  not  feasible  it's  not  important  and  has  no  payoff. 
10  cy  and  50  my  also  moke  me  question  feasibility. 


Not  easy.  Secret  is  to  segregate  process  requirements  from  equipment 
capabilities.  Then  generic  approach  is  feasible. 

Not  easy.  Secret  is  segregate  process  requirements  from  equipment 
capabilities.  Then  generic  approach  is  feasible. 

While  much  work  is  being  done  in  this  area,  it  is  sufficiently  complicated  to 
warrant  further  study.  Generative  process  planning  is  a  fertile  area  for 
research  and  development. 

I  think  the  scope  is  too  wide.  Specifically,  feature  extraction  should  be  a 
separate  project. 

Would  like  to  see  project  separated  into  more  definable  parts. 

Important  problem  -  I  feel  It  is  part  of  unit  process  activity. 

Scope  needs  redefining  -  must  carve  out  the  basic  research,  not  “the  whole 
enchalada'. 

Not  enough  detail  in  resource  estimate. 

A  lot  of  projects  are  in  progress  already  on  computer-aided  process 
planning.  A  clearer,  more  comprehensive  write-up  is  needed. 

Experience  in  building  this  type  of  system  has  taught  us  that  generative 
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planning  software  flexable  enough  to  apply  to  a  broad  range  of 
manufacturing  environments  would  be  so  unstructured  that  it  would  be 
practically  indistinguishable  from  commercial  At  development  software 
currently  available  or  soon  to  be  introduced. 

Needs  to  be  split  and  made  more  specific.  Apparently  applies  only  to 
fabrication  and  should  say  so. 

Still  to  general  and  global  to  be  accomplished. 

Time  and  resource  estimates  are  too  low  for  such  a  project. 

Project  2. 

The  objective  and  approach  says  absolutely  nothing!  The  design  of  a 
real-time  control  system  for  cell-level  manufacturing,  I  believe,  has  major 
payback,  i  feel  that  more  resources  should  be  devoted  to  this  project. 

I  feel  to  accomplish  the  project  objective  will  require  development  of 
learning  capability  which  is  a  definite  contribution  to  Al. 

Cl:  NBS &  Tl  are  currently  pursuing  it. 

C4:  Exception  handling  involves  context  dependent  diagnosis  and  planning 
integrated  together  in  a  near  real-time  environment.  No  work  has  been 
completed  so  the  issue  of  architecture  is  an  unanswered  question. 

Much  of  the  software  required  has  already  been  developed  (will  be  shown  at 
upcoming  IJCAI  in  LA.) 


Cl:  Approach  is  ill-defined,  how  augmented? 

C4:  Rule  based  controllers  are  in  production  use,  nothing  new. 

The  approach  and  milestones  do  not  give  a  picture  of  what  will  be  done. 
There  are  lots  of  buzz  words,  not  much  substance. 

As  a  basic  research  proposal,  there  is  no  statement  os  to  ‘new*  idea  being 
pursued.  There  exist  real-time  ES  systems  today  for  non-manufacturing 
applications.  Should  state  why  these  won't  work. 

In  our  FMS  environment,  the  exception  handling  capability  is  a  significant 
cost  driver.  This  is  not  basic  research,  it  is  advance  development  and  the 
project  should  take  only  3  years. 

Real-time  control  is  now  non-existent  (virtually)  in  manufacturing.  Basic 
research  is  extremely  important  in  this  area. 

There  are  many  kinds  of  cells.  Is  the  control  system  cell-specific,  or  con  a 
generic  control  system  be  developed? 

Tremendous  controversy  between  those  who  think  problem  is  solved  ond 
those  seeing  more  research  required.  No  consensus. 

Control  important  issue,  proposal  not  good. 


Might  include  example  of  type  of  rule  to  be  generated. 

For  this  and  the  other  scheduling  projects  (Projects  4  &  12)  my  comments 
are  the  same:  The  system  must  interface  to  the  shop  floor  equipment  in  a 
real-time  manner  through  the  use  of  sensors  and  status  feedback  to  the  cell 
Emphasis  needs  to  be  placed  on  the  parameters  needed  to  moke  decisions. 
Also,  the  scheduling  problem  should  be  decomposed  into  sub-goals  to  allow 
the  shop  to  perform  at  one  level,  the  cells  at  o  second  level,  and  the  wor* 
stations  ot  a  second  level.  We  don't  need  optimal  scheduling.  What  we  need 
is  a  schedule  that  is  feasible  ot  time  T.  The  schedule  con  be  changed  based 
on  the  environment  at  T+AT. 

Texas  A&M  University  is  under  contract  to  produce  a  proof  of  concept 
prototype  by  Christmas  which  controls  an  operating  FMS  for  a  large  Texas 
based  company. 

The  issue  of  At  based  control  for  due-date  driven  systems  has  never  been 
addressed  for  large  or  small  systems  and  what  architectures  are 
appropriate  is  an  open  question. 

Project  16. 

Response  does  not  address  this  issue. 

Rule  based  systems  are  too  shallow  to  produce  these  explanations 
effectively.  Some  sort  of  causal  model  or  qualitative  physics  approach 
would  be  better.  The  program  should  "understand"  the  origin  of  the  defects 
rather  than  just  Identify  them  by  rules. 
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I  think  8.5  my  is  too  much  effort. 

Cl:  Perhaps  feasible  in  electronics  manufacturing  or  for  a  rigidly  structured 
cell  environment,  but  determining  the  cause  of  defects  produced  in  a  shop 
with  manually  controlled  machines  by  looking  only  at  the  end  result  is 
ridiculous. 

C4:  There  ore  a  number  of  inductive  techniques  coming  on  the  open  market 
(i.e.,  EXPERT-EASE)  which  directly  address  this  problem.  Therefore,  I  don’t 
see  this  pushing  SO  A. 

C6:  Seems  well-estimated  to  me. 

Note:  the  ‘Achilles  Heel‘  of  this  project  is  that  it  assumes  the  existence  of 
a  large  amount  of  data.  Basically,  you  must  instrument  every  process 
parameter  capable  of  being  a  casual  factor  of  a  given  condition.  Data  of  this 
scope/quality  doesn't  ordinarily  exist  In  a  manufacturing  enterprise  (at 
least  not  now)...  big  practical  problem! 

Different  approach,  use  causal  model.  This  would  be  an  advance  in  the 
state-of-the-art  if  done  using  some  sort  of  causal  reasoning. 

This  had  the  highest  importance  rating.  In  a  meeting  with  8M,  Ford,  and 
Chrysler  engineers  about  18  months  ago,  a  very  similar  topic  was 
emphasized  as  important. 


Very  important,  fundamental  project  but  appears  to  fit  under  unit  processes. 


I  still  think  that  with  sufficient  Inspection  data  this  work  con  be 
accomplished  In  less  thon  6.5  MV. 

Great  idea,  but  I  think  some  application  in  Industry  is  possible  in  addition 
to  basic  research. 


Project  13. 

This  project  is  a  implementation  of  existing  concepts.  There  is  no 
additional  contribution  for  the  advancement  of  technology. 

Al  is  not  necessary  for  this  project. 

C4:  Sasic  research  on  purchasing  data  bases??  What  science  or  technology 
advance? 

1)  Purchasing/procurement  functions  ore  well  understood,  2)  many  of  the 
decision  support  elements  required  may  already  exist. 

Vague  approach. 

Design  of  experiments  is  a  excellent  application  area  for  capturing  existing 
layman  knowledge. 


C4:  Much  attention  has  historically  ben  given  to  procuring  parts  in  one's  own 


shop,  little  at  all  has  been  given  to  effectively  procuring  from  outside 
sources.  While  vendor  selection  performance  evaluation,  and  purchase 
activity  optimization  has  been  done  to  some  degree,  no  attempt  to  my 
knowledge,  has  been  made  to  intelligently  treat  vendors  os  resources  which 
need  to  be  evaluated,  selected,  managed,  and  controlled. 

In  our  (defense  electronics)  business  area,  this  is  a  tremendous  problem  - 
judicious  selection  of  of  materials  can  have  many  benefits  well  downstream 
of  purchasing  octivity! 

Possibly  the  greotest  payback  to  manufacturing  lies  with  this  family  of 
projects  for  material  strategic  and  tactical  planning.  Another  thrust 
appropriate  here. 

Not  basic  research  as  described.  Construction  would  be  to  place  the  global, 
strategic  problem  of  material  management. 

Good  problem  -  experts  exist.  Activities  possible  in  all  3  categories 
(research  -  MANTECH). 

It's  not  clear  how  or  whether  A1  would  help. 

1  really  think  this  system  has  merit,  especially  when  the  "global"  aspects  of 
outside  procurement  are  considered.  Remember,  too,  our  factories  are 
moving  rapidly  towards  increased  outside  sourcing. 


A  lot  of  this  kind  of  work  has  already  been  done.  Is  this  really  Al?  If  so, 
what  will  be  done? 


C3: 1  don’t  know  where  the  sensory  information  about  impending  failures 
will  come  from.  Sophisticated  multi-sensory  inspection  techniques  are 
difficult  issues,  but  assumed  here.  The  ES  may  not  have  many  symptoms  for 
its  data  base. 


The  project  is  incomplete.  It  must  also  analyze  sensor  data  to  take 
appropriate  action.  The  problem  is  better  suited  to  conventional  techniques 
other  than  Al. 

I  believe  this  belongs  to  the  unit  process  group. 


Cl  &  C2:  This  system  is  absolutely  necessary  for  successful  automation  of 
complex  systems. 

I  still  wonder  how  easy  it  is  to  ■detect'’  failure  across  a  broad  range  of 
applications.  This  capability  is  the  starting  point  of  the  project.  Can  we 
monitor  the  machine  only,  or  must  we  inspect  the  output  of  the  machine  - 
part  inspection,  a  highly  difficult  proposition. 


Good  project.  Important  but  belongs  under  the  preview  of  unit  processes. 


Building  an  Al  system  using  inputs  from  adaptively-controlled  machines  and 
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’expert’  reasons  is  really  a  good  idea,  but  I'm  not  sure  that  this  isn’t  a  basic 
research  issue. 


Very  good  proposal. 

Is  this  really  Al?  Consider  combining  with  Project  16. 

Group  3  Expert  Commentary.  The  following  are  the  top  five  Group  3 
projects,  and  the  associated  expert  commentary,  ordered  from  top  to 
bottom. 


Project  4. 

This  is  an  important  problem,  with  high  payoff  and  a  very  hard  problem.  I 
see  no  evidence  of  any  ideas  of  actually  how  to  work  the  problem.  Just  the 
statement  ‘extract  expert  knowledge ...  represent  it  in  experts  system,  etc." 
[sic]  Thus,  I  question  the  feasibility  of  the  proposed  effort. 

My  issue  with  Cl  is  with  the  use  of  group  technology  rather  than  other  more 
powerful  reasoning  techniques  for  information  retrieval  and  presentation  of 
manufacturability  advice. 

Project  2. 

The  cost  of  four  symbolics  is  sky  high.  Much  cheeper  computers  can  be  used. 

I  doubt  that  it  is  feasible  to  model  on  ’entire  manufacturing  capability’, 
such  os  aircraft  manufacturing  or  computer  manufacturing  in  3  years.  The 


project  needs  a  more  specific  focus.  I  doubt  that  there  is  'a  generic 
manufacturing  process". 

With  no  group  discussion  and  no  conflicting  data  from  the  computer  analysis 
I  can  find  no  reason  to  revise.  The  reason  for  the  importance  is  the  fact 
that  little  actual  scheduling  is  done  today  essentially  rule  based 
dispatching  is  what  is  performed.  I  believe  that  KB  techniques  applied  to 
scheduling  algorithm  selection  and  supervisor  level  is  important. 

I  feel  very  strongly  that  this  comprehensive  project  would,  like  HEARSAY, 
produce  significant  manufacturing  and  Al  contributions,  but  will  take  an 
effort  that  may  exceed  what  is  listed,  definitely  time  resources.  To 
produce  a  useful  result  would  take  a  large  effort,  at  least  more  than  3 
years,  I  believe. 

The  project  is  an  attempt  within  a  near-term  timeframe  to  solve  problems 
that  are  better  addressed  in  the  near-term  with  existing  approaches.Over 
the  long-term,  the  program  goals  make  since  -  but  an  exploratory  or  basic 
research  program  is  different  from  the  program  described.  This  program 
will  develop  models  and  techniques  that  no  one  will  use  in  real  factories. 
The  program  could  become  acceptable  with  a  reduced  level-of-effort,  longer 
timefrome  and  shift  to  exploratory  development  or  basic  research. 

Wasted  resources  ere  so  common  and  cost  so  much  that  the  importance  of 
getting  a  handle  on  them  cannot  be  over-emphasized. 


Project  6. 

The  problem  which  this  project  addresses  is  one  of  the  key  problems  which 
our  current  assessment  has  found  with  trying  to  use  existing  AI/ES 
development  tools  to  explore  manufacturing  problems. 

C2:  programming  language  project  -  attempts  to  design  a  data  base  using 
logic  ore  unrealistic. 

C3.  object  oriented  programming  area  is  well  developed  field  using  logic  for 
data  base  design  has  been  proposed  years  ago. 

C4: 1  don't  think  the  project  is  feasible  in  terms  of  data  base  design. 

This  is  a  very  key  project  but  also  very  difficult. 

Project  14. 

Adaptive  data  base  systems  are  a  good  idea  but  the  project  proposes  (on  a 
minimal  budget)  to  try  to  redo  the  liSS  and  multibase  technology 

-  no  new  or  real  A!  concepts. 

-  unrealistic  expectations. 

This  Is  a  very  important  topic  but  I  question  the  feasibility  of  solving  many 
problems  in  distributed  data  bases  in  3  years.  There  is  already  a  lot  of 
unsuccessful  work  in  this  area. 


Very  herd  problem.  Not  clear  new  approach  will  solve  the  basic  problem. 


I  feel  strongly  about  importance,  contribution  to  science,  and  payoff  of  this 
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project.  I  believe  this  would  revolutionize  our  capability  to  solve  many 
problems. 


Project  6. 

I  don’t  believe  the  project  has  a  great  contribution  to  moke  in  terms  of 
advancement  of  science. 

Unlike  frame  based  rule  driven  inferencing  techniques  GT  schemes 
essentially  freeze  knowledge  about  a  domain  into  one  efficient  but  rigid 
structure.  I  do  not  believe  that  development  of  another  GT  scheme  is  a 
relevant  Al  topic  nor  do  I  believe  that  an  approach  predicated  on  achieving 
an  optimal  GT  system  could  ever  converge. 


Appendix  B:  Criteria  heans/DSS  Input  Data 


Group  1  Project  Data:  Criteria  fleans/DSS  Input  Data 


Criteria* 

1 

2 

3 

4 

5 

ASD 

Project  * 
t 

5.60/56.0 

5.27/52.5 

5.40/54.0 

5.80/58.0 

4.47/44.5 

Std  Dev 

1.35 

1.16 

1.12 

1.47 

1.25 

6.35 

2 

5.20/52.0 

4.67/46.5 

4.47/44.5 

6.00/60.0 

5.47/54.5 

Std  Dev 

1.57 

1.84 

1.68 

1.13 

1.51 

7.73 

3 

5.00/50.0 

4.80/48.0 

5.13/51.5 

6.40/64.0 

4.00/40.0 

Std  Dev 

2.14 

2.21 

1.64 

1.92 

1.77 

9.68 

4 

7.00/70.0 

6.73/67.5 

6.67/66.5 

5.80/58.0 

4.13/41.5 

Std  Dev 

0.93 

1.22 

1.1 1 

1.52 

1.13 

5.91 

5 

6.73/67.5 

6.00/60.0 

6.33/63.5 

6.40/64.0 

4.53/45.5 

Std  Dev 

1.28 

1.25 

1.11 

1.59 

0.92 

6.15 

6 

7.47/74.5 

7.40/74.0 

7.40/74.0 

4.27/42.5 

4.33/43.5 

Std  Dev 

1.25 

1.24 

1.30 

1.44 

1.54 

6.77 

7 

6.00/60.0 

5.93/59.5 

5.87/58.5 

4. 07/40.5 

4.93/49.5 

Std  Dev 

2.07 

2.31 

2.33 

1.75 

2.31 

10.77 

8 

5.87/58.5 

5.73/57.5 

5.67/56.5 

5.67/56.5 

5.00/50.0 

Std  dev 

2.10 

2.34 

2.16 

2.02 

1.46 

10.08 

9 

4.53/45.5 

5.13/51.5 

5.60/56.0 

6.60/66.0 

3.27/32.5 

Std  Dev 

1.51 

1.41 

1.50 

2.16 

1.33 

7.91 

10 

7.20/72.0 

6.60/66.0 

6.67/66.5 

6.53/65.5 

3.53/35.5 

Std  Dev 

1.26 

1.24 

1.18 

1.19 

0.92 

5.79 

11 

5.53/55.5 

5.33/53.5 

5.73/57.5 

6.67/66.5 

3.53/35.5 

Std  Dev 

1.88 

1.76 

1.58 

1.36 

1.49 

8.07 

12 

7.13/71.5 

7.13/71.5 

6.67/66.5 

6.53/65.5 

3.67/36.5 

Srd  Dev 

1.13 

1.06 

1.05 

1.32 

1.11 

5.67 

13 

6.27/62.5 

6.53/65.5 

5.73/57.5 

3.93/39.5 

4.53/45.5 

Srd  Dev 

2.05 

1.88 

1.83 

1.71 

2.00 

9.47 

14 

7.53/75.5 

7.33/73.5 

7.27/72.5 

5.47/54.5 

4.27/42.5 

Std  Dev 

1.06 

0.98 

1.10 

1.60 

1.03 

5.77 

15 

3.93/39.5 

5.13/51.5 

5.00/50.0 

3.60/36.0 

3.47/34.5 

Std  Dev 

1.83 

1.25 

1.51 

1.12 

1.36 

7.07 

Group  2  Project  Doto:  Criteria  Means/DSS  Input 
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Criteria 

Group  2  Project  Data.  Criteria  Means/DSS  Input 

#  1  2  3  4  5 

6 

ASD 

$ 

Project 

1 

0 

6.71/67.0 

4.82/48.0 

5.14/51.5 

3.39/34.0 

3.93/39.5 

4. 07/40.5 

Std  Dev 

1.46 

1.39 

1.53 

1.31 

0.90 

1.12 

7.71 

2 

5.39/54.0 

7.39/74.0 

6.75/67.5 

6.50/65.0 

6.18/62.0 

5.21/52  0 

Std  Dev 

1.85 

1.47 

1.73 

1.77 

1.59 

1.34 

9.75 

3 

4.32/43.0 

6.68/67.0 

6.39/64.0 

4.14/41.5 

5.93/59.5 

6.50/65.0 

vJS 

Std  Dev 

1.79 

1.26 

1.40 

1.86 

1.21 

1.97 

9.51 

■Ms 

4 

4.96/49.5 

7.25/72.5 

7.00/70.0 

5.57/55.5 

4.64/46.5 

4.61/46.0 

Std  Dev 

2.12 

1.67 

1.25 

1.60 

1.25 

1.31 

9.20 

5 

6.21/62.0 

7.39/74.0 

7.21/72.0 

6.54/65.5 

6.36/63.5 

5.86/58.5 

Std  Dev 

1.95 

1.66 

1.83 

1.84 

1.45 

1.53 

10.26 

,.<v' 

6 

4.46/44.5 

6.29/63.0 

5.86/58.5 

5.79/58.0 

3.46/34.5 

3.36/33.5 

liTk 

Std  Dev 

2.10 

1.80 

1.65 

1.47 

1.32 

1.28 

962 

•w 

i?W 

7 

5.54/55.5 

6.36/63.5 

5.75/57.5 

5.68/57.0 

4.11/41.0 

4.39/44.0 

Std  Dev 

1.90 

1.25 

1.29 

1.70 

1.17 

0.92 

8.23 

8 

5.18/52.0 

5.50/55.0 

5.46/54.5 

3.79/38.0 

5.07/50.5 

5.18/52.0 

1  Std  Dev 

2.09 

1.75 

1.79 

1.79 

1.30 

1.52 

10.24 

9 

5.39/54.0 

6.04/60.5 

5.86/58.5 

6.39/64.0 

3.82/38.0 

4.54/45.5 

r-/- 

Std  Dev 

1.91 

2.01 

2.03 

1.47 

1.19 

1.20 

9.81 

10 

6.00/60.0 

6.64/66.5 

6.64/66.5 

5.43/54.5 

5.11/51.0 

5.00/50.0 

r-/ 

Std  Dev 

1.89 

1.16 

1.06 

1.37 

0.96 

1.09 

7.53 

11 

4.11/41. 0 

5.96/59.5 

5.96/59.5 

5.46/54.5 

3.68/37.0 

3.54/35.5 

Std  Dev 

1.99 

2.08 

2.01 

2.17 

1.54 

1.50 

11.29 

.*»» 

12 

5.11/51.0 

6.36/63.5 

6.25/62.5 

5.46/54.5 

4.89/49.0 

4.68/47.0 

jW{ 

Std  Dev 

1.81 

1.59 

1.51 

1.50 

1.26 

1.56 

9.23 

1® 

13 

5.89/59.0 

6.54/65.5 

6.32/63.0 

4.46/44.5 

5.46/54.5 

6.29/63.0 

iv|i 

Std  Dev 

1.62 

1.00 

0.96 

1.55 

0.74 

1.21 

7.10 

•3m 

14 

4.64/56.5 

5.68/57.0 

5.93/59.5 

5.43/54.5 

4.61/46.0 

4.57/45.5 

Std  Dev 

1.47 

1.54 

1.61 

1.43 

1.03 

1.26 

8.34 

15 

4.79/48.0 

4.64/46.5 

4.68/47.0 

4.32/43.0 

5.46/54.5 

5.39/54.0 

Std  Dev 

1.87 

1.54 

1.47 

1.54 

1.21 

1.23 

8.91 

IWO 

16 

6.36/63.5 

7.64/76.5 

7.21/72.0 

6.54/65.5 

4.36/43.5 

4.82/48.0 

Std  Dev 

1.50 

1.10 

1.42 

1.48 

0.91 

1.25 

7.66 

m 

17 

6.25/62.5 

6.68/67.0 

6.50/65.0 

5.50/55.0 

5.21/52.0 

4.71/47.0 

Std  Dev 

1.08 

0.98 

1.07 

1.37 

0.74 

0.90 

6.14 

t'>V. 

18 

6.21/62.0 

7.46/74.5 

7.29/73.0 

6.57/65.5 

3.96/39.5 

3.68/37.0 

m 

Std  Dev 

1.20 

1.20 

1.08 

1.32 

1.10 

1.28 

7.18 

19 

6.46/64.5 

7.14/71.5 

6.71/67.0 

6.54/65.5 

4.32/43.0 

4.43/44.5 

Std  DEv 

1.35 

1.48 

1.24 

1.71 

0.94 

0.92 

7.64 

20 

5.71/57.0 

6.82/68.0 

6.57/65.5 

5.82/58.0 

4.57/45.5 

5.82/58.0 

Std  Dev 

1.58 

1.28 

1.32 

1.68 

1.20 

1.61 

8.67 

I 
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Group  3  Project  Data:  Project  Meens/DSS  Input 


Criteria 

*  1 

2 

3 

4 

5 

6 

ASD 

Priject 4 
1 

9 

4.47/44.5 

5.60/56.0 

5.27/52.5 

4.53/45.5 

3.33/33.5 

4.20/42.0 

Std  Dev 

1.77 

2.16 

1.91 

2.29 

1.11 

1.66 

10.90 

2 

6.07/60.5 

6.93/69.5 

6.73/67.5 

6.73/67.5 

4.27/42.5 

5.07/50.5 

Std  Dev 

1.03 

1.53 

1.49 

1.71 

0.88 

1.53 

8.17 

3 

6.40/64.0 

6.67/66.5 

6.00/60.0 

6.47/64.5 

3.53/35.5 

3.27/32.5 

Std  Dev 

1.30 

1.45 

1.46 

1.30 

1.13 

0.88 

6.64 

4 

6.67/66.5 

7.67/76.5 

6.27/62.5 

7.20/72.0 

5.07/50.5 

5.27/52.5 

Std  Dev 

1.11 

0.90 

1.10 

1.01 

1.22 

1.39 

6.73 

5 

6.93/69.5 

4.67/46.5 

4.27/42.5 

4.60/46.0 

5.00/50.0 

5.40/54.0 

Std  Dev 

1.10 

1.80 

1.22 

1.64 

1.56 

1.45 

8.78 

6 

6.53/65.5 

6.33/63.5 

6.00/60.0 

6.60/66.0 

4.20/42.0 

4.00/40.0 

Std  Dev 

1,25 

1.23 

1.56 

1.12 

0.68 

0.85 

6.69 

7 

5.13/51.5 

5.87/58.5 

4.93/49.5 

5.20/52.0 

4.60/46.0 

3.80/38.0 

Std  Dev 

2.56 

2.23 

2.46 

2.48 

1.59 

1.47 

12.79 

8 

6.00/60.0 

6.67/66.5 

6.20/62.0 

6.33/63.5 

4.27/42.5 

4.60/46.0 

Std  Dev 

1.15 

1.63 

1.57 

1.63 

1.16 

1.06 

8.20 

9 

5.13/51.5 

6.47/64.5 

5.60/56.0 

5.67/56.5 

3.67/36.5 

3.27/32.5 

Std  Dev 

2.00 

1.96 

2.10 

2.53 

1.40 

1.16 

11.15 

10 

5.13/51.5 

5.27/52.5 

5.67/56.5 

4.73/47.5 

5.13/51.5 

5.13/51.5 

Std  Dev 

1.41 

2.05 

1.99 

1.96 

1.68 

1.51 

10.62 

11 

6.00/68.0 

6.87/60.5 

5.07/58.5 

6.60/66.0 

3.07/30.5 

3.53/35.5 

Std  Dev 

1.52 

1.06 

1.19 

1.59 

0.99 

1.19 

8.34 

12 

6.07/60.5 

6.33/63.5 

1.18 

5.27/52.5 

5.47/54.5 

4.73/47.5 

4.40/44.0 

Std  Dev 

1.33 

0.88 

1.36 

0.70 

0.99 

6.44 

13 

5.27/52.5 

5.93/59.5 

5.60/56.0 

5.33/53.5 

4.80/48.0 

5.27/52.5 

Std  Dev 

2.28 

2.31 

1.84 

2.13 

1.26 

1.62 

11.44 

14 

6.07/60.5 

6.93/69.5 

6.87/68.5 

6.40/64.0 

4.13/41.5 

3.87/38.5 

Std  Dev 

1.44 

2.02 

1.73 

2.03 

0.99 

1.19 

9.40 

Rounding  Error 

On  method  of  calculating  the  rounding  error  ie  as  follows.  If  a  number  b 
is  added  to  or  subtracted  from  the  raw  score  a  the  rounding  error  is: 


a  ±  b  -  a 


x  IOO  = 


±  1 00b 


% 


99 


If  c  is  added  to  or  subtracted  from  10a  the  rounding  error  is: 


10a  ±  c  -  1 0e 
10a 


x  1 00 


±  1 0c 


% 


Then: 


100b  _10c 
a  a 


or 

1 0b  >  c 


Therefore,  as  long  as  10b  >  c  is  true  the  rounding  error  for  the  entered 
numbers  will  be  less  than  the  error  than  if  they  were  rounded  and  entered 
without  multiplying  by  ten.  This  is  usually  the  case  since  the  worst  case 
for  the  raw  score  ends  in  x.25  (x  could  be  ony  integer)  and  would  be  rounded 
to  end  in  x.5,  while  multiplying  by  ten  would  result  in  the  entered  value 
having  no  rounding  error,  or  x2.5.  Only  when  the  raw  score  ended  in  x9, 
resulting  in  a  multiplied  value  of  x.9,  would  the  rounding  errors  be  equal. 


Appendix  C:  Management  Questionnaire  Results 


Question  Question  Score  Mean 

Number 


1 

5 

1 

5 

5 

4 

5 

5 

4 

5 

4 

4 

4.27 

2 

5 

4 

5 

5 

5 

5 

3 

5 

5 

5 

4 

4.64 

3 

5 

- 

5 

4 

5 

5 

5 

5 

4 

5 

5 

4.60 

4 

5 

4 

5 

4 

4 

3 

5 

4 

5 

4 

3 

4.18 

5 

5 

5 

5 

4 

4 

4 

5 

4 

4 

5 

5 

4.55 

6 

5 

5 

5 

5 

4 

4 

4 

5 

5 

4 

4 

4.55 

7 

5 

5 

5 

5 

3 

4 

4 

5 

5 

5 

5 

4.64 

6 

4 

3 

4 

4 

3 

5 

2 

4 

4 

5 

4 

3.82 

9 

4 

- 

3 

4 

3 

2 

2 

4 

4 

5 

4 

3.50 

10 

4 

5 

5 

4 

5 

4 

5 

5 

5 

5 

5 

4.73 

11 

5 

5 

5 

4 

5 

4 

5 

5 

5 

5 

5 

4.82 

12 

4 

5 

5 

4 

5 

4 

5 

5 

5 

5 

5 

4.73 

13 

5 

4 

5 

4 

5 

3 

4 

4 

4 

5 

3 

4.18 

14 

5 

4 

5 

4 

5 

3 

4 

4 

5 

5 

3 

4.27 

15 

5 

4 

5 

4 

4 

4 

4 

4 

4 

5 

4 

4.27 
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3c  *  fjw ww 


The  lost  question  on  the  questionnaire,  number  16,  asks  for  comments  or 
recommendations.  Not  every  manager  made  entries  for  question  16,  but 
those  that  were  made  are  listed  below. 

[Next  to  questions  13,  14,  and  15]  For  some  things. 

Nice  equipment. 

Good  presentation. 

Very  Important  component:  The  link  to  a  data  base  management  system. 
Very  important  activity:  The  selecting  and  structuring  of  the  decision 
model. 

Looks  like  great  tool. 

Should  be  used  more  instead  of  WAGs. 

Is  reproducible,  traceable  &  can  provide  accountability. 

Expansion  to  handle  more  than  5  units  at  a  time  would  be  very  desirable. 
Built-in  sub-routines  for  sensitivity  analysis  and  weighing 
trials/adjustments. 

“Mac"  environment  Is  excellent,  but  not  available  in  AFi 

Looks  Good. 

Main  ingredients  needed  to  have  this  kind  of  system  used  is  the  'getting  to 
know  the  Mac'  system.  In  current  environment  of  'command  driven*  IBMs 


this  represents  an  uphill  battle.  Also  I  feel  the  need  to  have  more  than  a  one 
hour  familiarization  time  with  such  a  system. 


1.  Emphasize  aid  -  peopls  make  decisions. 

4.  Real  world  more  human  interaction  -  less  structured. 

6.  More  so  the  decision  analysis 

better  tool  to  use  results  after  objectives  structured  - 
Defining  the  problem  makes  this  a  piece  of  cake. 

9.  This  does  not  prepare  data  it  only  manipulates  it.  Data  gathering  & 
preparation  hord  part. 


i 


i 


l 

I 
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